Optimizing Kafka Consumers with Kubernetes and KEDA...
summary
Starts with VPA → HPA on CPU/memory averages, then adds KEDA with Datadog scaler to trigger on Kafka consumer lag and lift average CPU use from ~15% to ~57% while holding data‑freshness SLAs and cutting daily cost ~55%. Builds an in‑house CRD called resource advisor that runs off‑peak, gathers historical Datadog metrics, recommends vertical resizes and applies them to handle organic traffic shifts; system works but still faces uneven pod load from partition assignment and plans rate limiting and richer custom metrics.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:56Platform goals
- 5:38First attempt with VPA
- 8:50Switching to Horizontal Pod Autoscaler
- 11:16Challenges with Horizontal Pod Autoscaler
- 13:21KEDA for scaling
- 16:53Resource Advisor operator
- 20:47Benefits and next steps
- 23:19Q&A
related talks
34mAutoscaling Made Easy with Kubernetes Event-Driven Autoscaling (KEDA)
13mScaling Kafka on Kubernetes
37mKEDA: Unlocking Advanced Event-Driven Scaling for Kubernetes
38mScaling EDA Workloads with Kubernetes, KEDA & Karpenter
49mEvent-driven autoscaling for Serverless Java
46mScaling EDA Workloads with Kubernetes, KEDA & Karpenter
36mAutoscaling Isn't Set-and-Forget: Tuning KEDA & Karpenter for Real Workloads.
47mKEDA, HPA and VPA? An Introduction to Scaling