Join us for a live webinar
Beyond Load Balancers: Scaling Out GenAI with Message Queues on Kubernetes
Wednesday, September 25th 2024, 12pm EST.
Load balancers are a staple of scalable, high-throughput, high-availability architectures. They work great to scale web services. When requests take longer, though, things get complicated. Requests can pile up on some backends; bursts of traffic can send the latency through the roof; and when autoscaling kicks in, it might be too late and/or too expensive.
Asynchronous architectures and message queues can help a lot here combined with event-driven autoscaling.
Register Now
We're going to see how to implement that pattern on Kubernetes, leveraging:
- A popular LLM to generate thousands of completions;
- RabbitMQ and PostgreSQL to store requests and responses;
- Bento to implement API servers, producers, and consumers without writing code;
- Prometheus, Grafana, and KEDA for observability, dashboard, and autoscaling;
- Helm and Helmfile to automate deployment as much as possible.
Who should join:
- DevOps, Platform, and SRE professionals looking for ways to improve their autoscaling practices.
- Data engineers who want a better understanding of running their workloads on Kubernetes.
Meet our Experts
.png?width=175&height=175&name=heavybit-square%201%20(2).png "heavybit-square 1 (2)")
Jerome Petazzoni
Tinkerer Extraordinaire
Part of the Docker founding team. Docker Community Advocate between 2013 and 2018. These days he teaches Kubernetes at Enix, a French Cloud Native shop.
When he's not busy with computers, he collects musical instruments, and can arguably play the theme of Zelda on a dozen of them.
.png?width=175&height=175&name=image%20252%20(1).png "Anton Weiss")
Anton Weiss
Chief Cluster Whisperer, PerfectScale
Software delivery optimization expert and Kubernetes fanboy. With previous experience as a CD Unit Leader, Head of DevOps, CTO, and CEO he has worn many hats as a consultant, instructor, and public speaker.
He is passionate about leveraging his expertise to support the needs of DevOps, Platform Engineering, and Kubernetes communities.