Building and Operating a Pretty Big Storage System (My Adventures in Amazon S3)
summary
Describes building S3 at scale with concrete numbers and engineering choices: 31 regions and 99 AZs, ~280 trillion objects, 100M requests/sec and 400 Tb/s peak, and ~4 billion integrity checks/sec. Uses Shard Store in Rust with log structured layouts and soft updates, moves from eventual to full consistency, and relies on erasure coding plus spreading data across tens of thousands of drives per customer to handle 2.3M req/sec bursts. Durability uses an 11 nines durability model, formal verification in Rust, staged shard rollouts and durability reviews — practices that prove effective in production.
AI-written summary · about
chapters (11)
- 0:00Introduction and welcome
- 0:28Career path and milestones
- 5:56Inside the S3 architecture
- 10:48S3 scale and customer impact
- 14:09Hard drive fleet management
- 21:18Heat management and data placement
- 29:30Durability and new technologies
- 37:27Deployment and verification
- 40:00Customer data protection
- 42:35Scaling myself and organization
- 46:44Ownership and academic bridges
related talks
33mPredicting the Future of Distributed Systems
56mMastering Cloud Scalability and System Designs Deep dive
41mDesign (Failures) Case Study
32mScaling Done Right
37mSharding: Growing Systems from Node-scale to Planet-scale
59mMastering the System Design Methodology
36mDesigning for failure - architecting fault-tolerant system
17mExtreme Engineering-Elements of Building a High -scale Broking Application