Building Container Images In Kubernetes: It’s Been a Journey!
summary
Datadog migrates image builds into Kubernetes using buildkit in rootless mode, achieving native multi-arch builds and retiring Docker runners; over 90% of images build fine but ~10% expose tricky kernel and filesystem interactions. Issues include failing to set SELinux xattrs, leaked background processes because rootless disables process sandboxes, and overlayfs masking bugs caused by opaque tombstones fixed only after kernel 5.11’s user exact option; result: several hundred images built on-cluster and >1,000 builds/day reliably.
AI-written summary · about
chapters (21)
- 0:00Start
- 1:43History: Building docker images
- 3:03History: Building ARM binaries
- 3:30History: Building multiarch images
- 4:19Limits of this system
- 5:52Can we use them?
- 8:49Overview
- 9:15Example
- 10:33A complex Dockerfile
- 13:22A more complex Dockerfile
- 14:46Let's debug!
- 15:47Basic Reproducer
- 17:39How does it work usually?
- 18:13proc in containers
- 19:38Conclusion
- 22:50We use Overlayfs. How does it work?
- 25:16What about step #3?
- 26:05Taking a step back
- 27:42Opaque directory implementation
- 28:57Mysteries
- 29:21Kernel function tracing
related talks
43mBuilding Docker Images on Kubernetes: A Long Journey!
29mBuilding Images Efficiently and Securely on Kubernetes with BuildKit
35mBuildKit: A Modern Builder Toolkit on Top of containerd
20mBuildKit CLI for kubectl: A New Way to Build Container Images
25mRunning Cache-Efficient Builds at Scale on Kubernetes with BuildKit
34mUnprivileged Image Builds: What are the Challenges and Where are we Today?
46mA Security State of Mind: Continuous Security with Kubernetes
46mAround & After Kubernetes: The Principles and Ideas that Guide Us