Building Docker Images on Kubernetes: A Long Journey!
summary
Datadog describes moving Docker image builds into Kubernetes using BuildKit rootless and buildx with QEMU for multi-arch, replacing Docker-in-Docker runners. Across hundreds of images and thousands of daily builds this yields 90–95% immediate success and reduces ARM multi-arch build time from ~4 hours to 10–15 minutes. Remaining 5–10% failures are traced to concrete issues: extended xattrs blocked in user namespaces, disabled process sandboxing causing leaked background processes/zombies, and overlayfs opaque-xattr caching bugs (fixed in kernel 5.11+ or distro patches), with some security tradeoffs and imperfect workarounds.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 2:00Datadog Overview
- 3:25Datadog History
- 6:25Problems Encountered
- 7:25Solutions
- 13:40First Problem
- 17:04Second Problem
- 24:08Third Problem
- 30:14Fourth Problem
- 32:06Fifth Problem
- 37:25Conclusion
related talks
35mBuilding Container Images In Kubernetes: It’s Been a Journey!
29mBuilding Images Efficiently and Securely on Kubernetes with BuildKit
35mBuildKit: A Modern Builder Toolkit on Top of containerd
29mHow to Improve Your Image Builds Using Advance Docker Build
28mWhat's New in Docker Build
26mContainerization crossroads: Experiences-Experiments-Expectations
20mBuildKit CLI for kubectl: A New Way to Build Container Images
39mAbusing Docker builds with BuildKit and friends