Googlies! Grubbers! Lollies! How cricket helped us improve our throughput by 200%
summary
Traffic spikes from Hotstar streaming IPL push a real-time HTTP request sender from ~20M to ~40M events/min, causing Kafka consumer lag; profiling with flame graphs and perf reveals SSL CPU hot spots inside HTTP kit and backpressure from internal queues. Fixes: increase concurrency (more producer threads), add a second HTTP client instance, reconfigure Kafka partitions and Docker instances. Result: per-instance throughput rises 200%, fleet size drops to one-half–one-third, and the pipeline handles subsequent spikes like IPL, Black Friday and Singles Day reliably under Little’s law guidance.
AI-written summary · about
chapters (34)
- 0:00Introduction
- 0:52What is AppsFire
- 1:47Why are we here
- 3:03Cricket
- 3:58Hotstar
- 4:15Events
- 4:42Services
- 5:31Consumer Lag
- 6:01Kafka Partitions
- 6:22Upgrading Instance Types
- 7:36The Jesus
- 7:58Littles Law
- 8:50Everything is Little
- 9:16Variables
- 10:01Application
- 13:16Queue latency
- 13:58Latency
- 14:57Capacity Increase
- 15:13Kafka Subsystems
- 17:13What is a bottleneck
- 17:58Technical instances
- 18:11Adding concurrency
- 19:03Flame graphs
- 19:35HTTP Kit
- 20:14SSL
- 20:45HTTPClient
- 21:41The bottleneck
- 22:46The results
- 23:28The recipe
- 24:05Applications
- 24:34Milton Friedman
- 25:17Summary
- 26:02Mumbai Indians
- 26:16Chennai Super Kings
related talks
18m"Kafka's Metamorphosis"
53mDemystifying Kafka
38mKafka Message Broker Under High Load
26mHow We Prepped Our Infrastructure for a Super-Bowl Sized Traffic
33mReaching Ludicrous Speed
39m400 Billion Messages a Day with Kafka
24mPushing Your Streaming Platform to the Limit
41mHandling billions of events per day with Kotlin Coroutines