Faster Container Image Distribution on a Variety of Tools with Lazy Pulling
Talkat KubeCon + CloudNativeCon North America 2021 · on Thursday, 2021-10-14· at 11:55· Room 408 AB + Online·
summary
Pulling causes 76% of container start time while only 6.4% of bytes are read, so lazy pulling downloads needed file chunks on demand instead of the whole image. Two implementations exist: stargz (backward-compatible extension with prefetched “prioritized files” and a stargz snapshotter for containerd/CRI-O/Podman/BuildKit/k3s) and nydus (FUSE daemon, chunk deduplication, end-to-end integrity, used at Alibaba with Dragonfly p2p). Benchmarks show workflows drop ~94s→52s and simple starts drop ~25–32s→~5s; both work in production.
AI-written summary · about
chapters (26)
- 0:00Intro
- 0:24Pull is time-consuming
- 3:15eStargz: Standard-compatible lazy pulling
- 4:11e Stargz image layer format
- 5:28Benchmarking result
- 6:23Lazy pulling on Kubernetes
- 8:41Lazy pulling on containerd
- 10:04Lazy pulling on Podman/CRI-O
- 11:35Lazy pulling on Build Kit
- 12:24Building eStargz
- 13:05Converting an image into eStargz
- 14:21Discussion toward OClv1 extension
- 15:22Nydus: Improved Lazy Pulling and More
- 18:59FUSE/virtiofs Unified Architecture
- 20:09Registry Acceleration File System
- 20:45OCI Spec Compatibility
- 21:33Lazy Pulling Benchmark
- 22:09Lazy Pulling with Variety of Tools
- 22:50Manual Image Conversion -- nydusify
- 23:29Manual Image Conversion -- buildkit
- 23:57Automatic Conversion with Harbor
- 25:00OCI Artifacts Manifest
- 26:17Community V2 Image Spec Requirements
- 27:45Recap
- 28:53eStargz Project Information
- 29:21Nydus Project Information
related talks
34mThree Approaches to Speed up Image Distribution in Cloud Native Era
34mFast Image Pulls Using IPFS And Opportunistic Caching
40mNode-Level Optimization in Large Cluster Powered by Containerd Plugin
14mLightning Talk: How to Get Slim with Soci and Stars: Minifying Containers with Lazy Image Loading in ContainerD
36mP2P Docker Image Distribution in Hybrid Cloud Environment with Kraken
49mContainers All the Way Down!
16mSimplifying the Journey of Your Containerized Application from Dev to Production
25mStop Re-Downloading Your Container Images: Content-Based Layers with Chunkah