Maturing Your Data Architecture in a Week: How Bluesky Survived
summary
Bluesky scales 10x in weeks to 33M users and peaks near 50k API requests/s and ~1,500 write events/s on bare‑metal across two data centers, forcing an 11‑day, 16‑hour‑a‑day emergency. Engineers debug a 700 GB core dump causing cluster-wide throttling, hit NVMe garbage‑collection limits that exhaust SSD overprovisioning, and replace a crashing single‑node relay with a non-archival relay plus offloaded mirrors. A dynamic celebrity cache cuts profile query P99s from ~100ms to ~10ms — a 90% reduction in query P99s and ~50% fewer peak queries.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 1:44Scaling challenges overview
- 5:58Modes of failure
- 8:25Data center outage incident
- 11:33Database stability issues
- 15:41Firehose relay scaling
- 23:11The Ouroboros situation
- 27:08Celebrity caching strategy
- 30:49Invalid handles and DNS
- 33:50Additional service incidents
- 37:19Recovery and conclusion
related talks
25mData Demons: Unleashing Chaos or Driving Performance?
01h 06mBest practices when accessing Big Data or any other data!
29mBlueprints for Scaling: Practical Steps for Scaling Your Data Platform to Handle Exponential Growth
42mBillion-Scale Triumph: Innovating at the Edge of Cloud Data
41mOn Data: Sky above, sand below, peace within
43mArchitecting for Speed & Scale
31mFrom Raw Data to Real Impact: Lessons from Real-World Transformations
30mInfrastructure scale issues