Speeding Up Analysis Pipelines with Remote Container Images
Talkat KubeCon + CloudNativeCon North America 2020 Virtual · on Thursday, 2020-11-19· at 16:50· Intrado Virtual Event Platform,·
summary
CERN describes using containerd remote snapshotter and estargz to lazy‑load huge analysis images (an atlas/athena release ~17.2 GB uncompressed, 5.4 GB compressed). Startup becomes flat ~5–6 seconds versus ~3.3 minutes for a full pull, and network egress drops from ~6 GB to <1 GB for a job, with runtime overhead ≈+1 minute; building an estargz‑optimized image takes ~45 minutes. Deployment uses CVMFS‑style hierarchies or replicated Harbor registries and a demo runs hundreds of parallel jobs showing the savings.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:15What is CERN
- 4:25CERN VMFS
- 5:35Container Efficiency
- 7:11History
- 7:54Lazy Pulling
- 11:44Experiments
- 14:20Demonstration
- 17:39Software Distribution
- 20:16Deployment Overview
- 21:48Workflow Overview
- 25:05Network Traffic
- 26:14Snapshotter Status
- 27:35Improvements
- 29:03Closing remarks
related talks
29mImage Snapshotters for Efficient Container Execution in Particle Physics
30mWhen Things Get Big
10mBuilding 1300 Container Images in 4 Minutes
27mOptimizing CI/CD Pipelines for Faster Time to Market
49mThe Quest for the Fastest Deployment Time
11mFast and Efficient Container Startup at the Edge via Dependency Scheduling
47mCloud Native and Sustainable, Reproducible Scientific Computing
25mBlazing-Fast Container Deployments: Image Caching With Block Storage