Object Compaction in Cloud for High Yield
summary
Compacts thousands of tiny files (tens of bytes–tens of KB) into large write-sized blobs to avoid per-PUT throttling and reduce request overhead, using client-side packing and an in-blob index that stores offsets and small tombstones. Reads use ranged GETs against blob offsets; deletes mark entries and trigger background blob recycling that rewrites live entries into new blobs. Implementation includes per-blob metadata objects and a reclamation policy based on dead-bytes fraction; expected result is far fewer PUTs and lower cost/latency proportional to the packing ratio (e.g., 1,000 small objects → 1 blob).
AI-written summary · about
chapters (6)
- 0:00Introduction to object compaction
- 2:58Problem with small files in cloud
- 5:52Cost benefits of compaction
- 8:43Policy and naming conventions
- 12:16Data deletion and archiving
- 15:28Q&A and conclusion
related talks
14mLeveraging software optimization to reduce cloud footprint
17mBoosting Cost Efficiency by Cost Aware Architecture
25mOptimizing Container Registry Storage: Introducing Blob De-Duplication for Accurate Sizing and Cost
25mStreamlined Efficiency: Unshackling Kubernetes Image Volumes for Rapid AI Model...
27mUnlocking Storage Efficiency: Diagnosing & Optimizing Performance in Scaling Infra
24mEfficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache Iceberg
23mOn-demand Container Loading in AWS Lambda
16mPolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases