Exploring Traffic Jams in Your Data Flows
summary
Shows that producer-dispatcher-consumer model plus an intermittently waking dispatcher creates phantom jams: a Seastar/Scylla simulator with virtual time, SSD-like consumer at 200k req/s consumer, and dispatcher tuned to 1.5x load collapses once input exceeds about 160k req/s threshold, pushing latencies from milliseconds to tens or hundreds of seconds. Measured on a 4-node cluster (writes ~150 MB/s vs expected ~170). Fixes tried: larger batches, dedicating cores, or sending everything (helps throughput but loses scheduling and can worsen tail latency).
AI-written summary · about
chapters (10)
- 0:00Welcome
- 8:14Speaker background
- 12:54Phantom traffic jams
- 21:48Producer-dispatcher-consumer model
- 31:01Live demo with ScyllaDB
- 37:31Why phantom jams occur
- 41:02Monitoring value
- 46:25Q&A: adaptive dispatchers
- 55:42Q&A: diagnosing dispatcher issues
- 59:04Summary
related talks
41mPushing RabbitMQ to the limit
30mQueueing Theory, In Practice: Performance Modelling in Cloud-Native Territory [I]
49mResponding in a Timely Manner
38mPractical Performance Analysis
46mQueueing Theory in Practice: Performance Modeling for the Working Engineer
18mMeasuring Congestion in High Performance Datacenter Interconnects
33mReaching Ludicrous Speed
30mThe Gashlycrumb Tinies of AI Networking You Must Know (or Languish!)