Designing Features for Mature Systems: Lessons Learned from Manta
summary
Implements S3-style multipart uploads for Manta by creating per-upload UUIDs, storing parts as normal objects under uploads/ with prefix directories to avoid directory limits, and choosing target storage nodes up front so parts are co-located. Adds a storage-side operation Mako finalize to stitch parts into a flat file, uses MD5+size checks for idempotency, and records a finalizing record on the target object shard so commits are atomic across metadata and storage. Defaults to two replicas; supports up to 10,000 parts and works in production.
AI-written summary · about
chapters (22)
- 0:00Intro
- 2:31Manta 101: Features
- 4:20Manta 101: API operations
- 4:52Manta 101: System Primitives
- 5:45Manta 101: Design Constraints
- 7:55Manta Design: Data Path
- 13:15Manta Design: Compute
- 14:25Multipart Uploads!
- 15:07Multipart Uploads: First Principles
- 16:28Multipart Uploads: Design Questions
- 17:12Multipart Uploads: Design Considerations
- 21:23Example MPU Structure
- 22:02Multipart Upload Design: Commits
- 24:26Commit: Storage Layer Implementation
- 30:45Multipart Upload Commits: Idempotency
- 34:34MPU State Machine
- 35:15Commits: Final Steps
- 35:30Concurrency in other MPU operations
- 36:05Revisiting Design Constraints
- 37:16Working with Design Constraints
- 38:03Tradeoffs in Design
- 39:49Final Thoughts
related talks
53mBuilding and Operating a Pretty Big Storage System (My Adventures in Amazon S3)
01h 03mArchitecting a large social network without breaking the bank
19mBreaking Storage Barriers: How RabbitMQ Streams Scale Beyond Local Disk
30mDistributed system design
58mAnchored metadata
32mElevating Scalable Object Storage: A Comprehensive Exploration of Ozone’s Trailblazing Capabilities
13mExplaining Distributed Systems Like I'm 5
38mRe-architecting Apache Kafka for the Cloud