Petabyte Scale Object Storage Service Using Ceph in A Private Cloud
summary
Flipkart runs a Ceph S3-compatible object service in two DCs, claiming production use for ~3 years and a single cluster at 2 PB raw, 750 OSDs, 1.5B objects that grows ~50–100M objects/month. Architecture uses HAProxy → nginx → read/write RadosGW pools, SSD journals and an SSD writeback cache, OpenTSDB/Graphana monitoring and alerts. Tough spots include rebalancing and metadata-pool outages, multipart races and leaked objects, painful Hammer→Jewel→Luminous/BlueStore upgrades; mitigation is sharding, offline recovery tools and a per-user rate limiter embedded in radosgw, with some manual workarounds still required.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 4:42Overview of Petabyte
- 5:25Tech Stack
- 6:46Virtual Infrastructure
- 7:21Clients
- 8:48Instances
- 10:06QoS
- 11:58Health Monitoring
- 13:56Request Monitoring
- 14:37Operations Monitoring
- 15:16Challenges
- 16:53Metadata Pool
- 18:37Application Error
- 19:27Missing Features
- 20:17Multipart Loss
- 21:35Cluster Challenges
- 23:35Missing Pieces
related talks
06mPractices of Ceph Object Storage in Public Cloud Services
33mEnhancing Ceph for Infinite Scalability at LINE
41mBeyond Particle Physics: The Impact of Ceph and OpenStack on CERN's Multi-D...
21mCeph for Public Cloud Workloads
49mCeph BlueStore — a New Storage Backend in Ceph
38mBringing a Ceph Based Enterprise Email System Into the Field
30mDeploying and Operating CephFS for Scientific Applications at Fermilab
05mThe Enterprise Ceph Spectrum