Optimizing Container Registry Storage: Introducing Blob De-Duplication for Accurate Sizing and Cost
summary
Quay implements blob de-duplication to report true storage per namespace and repository by splitting work into a backfill and running total. Database sizes are large — about 260,000 namespaces, ~700,000 repositories and 100M+ manifest_blob rows — so initial SUM queries take >30s and fail. The final design checks blob membership on each push/delete (adds ~10 ms average), runs a single backfill pod (completed in ~2 days vs estimated 5–7), uses read replicas and a missing-index fix, and operates at scale.
AI-written summary · about
chapters (14)
- 0:00Quay storage problem
- 2:11Initial sizing limits
- 5:28Accurate sizing at scale
- 7:13Solution development
- 10:03Background workers
- 11:06Layer deduplication
- 13:10New sizing implementation
- 15:01On-prem deployment
- 16:26Quay.io deployment
- 18:12Backfill worker
- 18:53API performance fixes
- 19:37Results and insights
- 21:24Lessons and future
- 23:12Q&A on reference counting
related talks
27mUnlocking Storage Efficiency: Diagnosing & Optimizing Performance in Scaling Infra
19mObject Compaction in Cloud for High Yield
14mInftyDedup: Scalable and Cost-Effective Cloud Tiering with Deduplication
25mStreamlined Efficiency: Unshackling Kubernetes Image Volumes for Rapid AI Model...
44mOpenSearch’s Quest for Disaggregated Storage: The Strangler Fig Patter...
10mBuilding 1300 Container Images in 4 Minutes
30mFaster Container Image Distribution on a Variety of Tools with Lazy Pulling
17mBoosting Cost Efficiency by Cost Aware Architecture