Enhancing Ceph for Infinite Scalability at LINE
summary
LINE builds a multi-backend object service that fronts multiple Ceph subclusters and a cost‑efficient S3 store (Dragon) behind a single endpoint, operating 20+ subclusters and 700+ PB. It adds a router abstraction for bucket placement, request routing, cross‑cluster CopyObject, and QoS; a migration API that preserves metadata and moves SSE‑C objects without decryption (adds AES256‑CTR support); and a customized migrator based on Arclone with Kafka, MongoDB and FaaS hooks. Prototype works end‑to‑end; candidate selection targets buckets ~2–3 months old and production tuning is underway.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:03Background and scalability needs
- 4:39Introducing the router layer
- 8:58QoS and hot data patterns
- 12:09Migration target selection
- 12:31Migration API design
- 21:28Migrator implementation
- 25:54Future contributions
- 26:15Q&A session
related talks
26mPetabyte Scale Object Storage Service Using Ceph in A Private Cloud
06mPractices of Ceph Object Storage in Public Cloud Services
41mBeyond Particle Physics: The Impact of Ceph and OpenStack on CERN's Multi-D...
38mBringing a Ceph Based Enterprise Email System Into the Field
21mCeph for Public Cloud Workloads
05mThe Enterprise Ceph Spectrum
32mDeploying Stand-alone Ceph and Integrating with K8/OpenShift and OpenStack
19mKeynote: Operating 600+ Mesos Servers on 7 Datacenters @Criteo