Containers are hideously undebuggable black boxes and we never should have invented them
summary
Ceph migrates packaged daemons to containers (Podman/Docker/Kubernetes) using cephadm and Rook, which eases deployment but adds indirection: troubleshooting requires running generated systemd unit run scripts, invoking Podman run, and checking the journal instead of /var/log. During Nautilus→Octopus upgrades two of six OSDs fail to start because the generated unit binds /var/run/ceph before the host dir exists (error: var/run/ceph fsid no such file or directory). Useful commands: cephadm shell, ceph orch ps; registry misconfiguration can pull mixed image versions.
AI-written summary · about
chapters (5)
- 0:00Start
- 0:17CONTAINERS ARE HIDEOUSLY UNDEBUGGABLE BLACK BOXES AND WE NEVER SHOULD HAVE INVENTED THEM
- 28:32CONTAINER ORCHESTRATION
- 35:00USABILITY VS. COMPLEXITY
- 36:53QUESTIONS?
related talks
29mTroubleshooting a Cephadm Deployment
35mCritical Troubleshooting Tools for Rook Clusters
44mContainer Management at Scale
01h 39mLab: Linux Container Internals
05mHealthier Ceph Clusters with Ceph-medic
12mUsing rbd Migration Tooling in Our Ceph Operations
44mThe Containerization of the Operating System: Exploring Bootc and the Future of Linux Deployment
43mIntro: containerd
