P99 Pursuit: 8 Years of Battling P99
summary
ScyllaDB pursues sub-ms P99 by building shard-per-core two-level sharding and CPU and IO schedulers that tag every tiny operation, plus admission control and a block detector that finds memory allocator stalls. Engineering moves include contributions to Linux XFS, pinning threads/containers, and AWS i4i benchmarks showing 180 TB single-node runs and 256‑core clusters; cluster doubling happens in minutes. Those changes cut extreme tails (before/after latency graphs go from spiky to flat) and reveal 22x P99 hits when a slow resource becomes the bottleneck.
AI-written summary · about
chapters (5)
- 0:00Introduction and company background
- 3:19Sila DB features and architecture
- 8:23P99 pursuit and latency challenges
- 13:59Sources of latency and solutions
- 16:30Summary and Q&A
related talks
16mTowards Optimal Rack-scale μs-level CPU Scheduling through In-Network Workload...
45mBeyond Machine Boundaries: Experience with Memory Disaggregation
39mThrottling: New Developments in Application Performance with CPU Limits
51mThe Quest for Speed: Journey to 50% Better P99 Times with Go
46mTaming the 9s
01h 06mBest practices when accessing Big Data or any other data!
32mSharding for mission-critical, internet-scale apps
27mTales at Scale: Analytics at 1000 QPS and Beyond