Breaking the 125µs @ 1Mhz barrier w/ Tokio and what it means for your Tokio Web Server
summary
Claim: a Tokio-based proxy at Momento hits 125µs @ 1MHz P99 by combining batching, core affinity and kernel tuning. The team builds a high‑perf proxy (fronting Valkey and DynamoDB), adds level runtime with socket reuseport, pins IRQs to dedicated cores, and uses batching plus load‑shed to avoid busiest‑core tails. Results reach ~1M req/s on Graviton4 in synthetic tests; instrumentation via Resilous and Valkyrie Lab exposes where it breaks under real‑world connection imbalance.
AI-written summary · about
chapters (21)
- 0:00Tokio limits from Mars rovers
- 1:16CS education lied about big O
- 3:45Intimate system knowledge matters
- 4:29Proxy bottleneck in caching
- 7:20High-performance Tokio proxy
- 9:04Breaking 125µs barrier with Tokio
- 11:56Level runtime for core pinning
- 13:19Benchmark results with level runtime
- 14:22Load imbalance challenges
- 17:51Tokio runtime configuration knobs
- 19:15Parameter sweeps in tuning
- 20:39Burn configuration and core retention
- 23:26Benchmarking tools Valkyrie Lab Bristleless
- 25:51Instrumentation for root cause analysis
- 26:35Live demo Resilous performance
- 30:25Latency improvements with level runtime
- 31:51Batching and syscall efficiency
- 33:16Deploying Resilous for production
- 34:40Comparing experiments with Resilous
- 35:42Low-latency system design takeaways
- 36:03Summary
related talks
33mReaching Ludicrous Speed
48mBetter performances on the server side with HTTP/2
32mTechniques for writing concurrent applications with asynchronous I/O
05m“Using async-profiler to Find Tail Latency Issues”
26mAre You Sure About Your Mesh Performance?
46mReal-world HTTP benchmarking
38mPractical Performance Analysis
53mComet, Ajax, web browsers limitations and web apps performance factors