Predictive Load-Balancing: Unfair but Faster & more Robust
summary
Predictive load balancing claims to cut tail latency by treating per-request elapsed time as offered load and using a streaming median over a moving window to predict server slowness. It combines queuing theory’s instantaneous traffic estimate with a utility-style penalty for error rates, uses a probation/warmup state for new servers, and sub-selects small server views to keep stats fresh. Simulations and Netflix experiments show p99 falling from ~15s to ~5s in demos, it detects GC pauses within 1–2 median latencies, and one Twitter service runs it in production.
AI-written summary · about
chapters (11)
- 0:00Start
- 0:27What is load balancing?
- 9:03Load-balancing implies "balancing"
- 16:28Load-Balancing Matrix
- 17:11Latency based Load-Balancing
- 19:08How to measure the latency?
- 20:49Recency is important
- 24:38What if latency estimation is off?
- 25:27Fast server but Bad server
- 26:25Keeping fresh statistics
- 41:43Conclusion
related talks
21mHandling Traffic Spikes Through Automated Prioritized Load Shedding
34mPredictive preloads for ultimate web performance?
41mLoad Balancing Roundup
30mQueueing Theory, In Practice: Performance Modelling in Cloud-Native Territory [I]
38mA Practical Look at Performance Theory
19mFlash Sale Engineering
22mLoad Balancing in gRPC
46mQueueing Theory in Practice: Performance Modeling for the Working Engineer