Nightmares of a Container Orchestration System
summary
Practical failures and fixes for Mesos/DC/OS clusters: require immutable, versioned images and a private registry to avoid silent changes; monitor disk and implement Docker garbage collection; measure resource slack (allocated minus used) and set realistic CPU/memory requests; beware Java pre‑9 JVMs ignoring cgroups so many JVMs on a big host swamp CPU and memory; use the Mesos executor model to upgrade agents without killing tasks; service mesh sidecars like Envoy sidecar add features but cost L7 throughput — prefer L4 when possible; run Zookeeper as a 5-node quorum.
AI-written summary · about
chapters (33)
- 0:00Intro
- 1:44Scope of Jan's nightmares...
- 2:42Container (wo Orchestration)
- 3:35Container Orchestration
- 6:47Backup State
- 8:15Networking
- 13:26Immutable Container Images
- 14:40Private Container Registries
- 15:19Repeatable Container Builds
- 17:59Disk Usage
- 19:12Resource Constraints
- 20:58Zookeeper Cluster Size
- 22:22Health Checks
- 24:55NoSQL Datastores
- 26:08Removing Stateful Frameworks
- 27:09Container vs Container
- 30:23Java meets Container
- 31:36a.ka. Understand Container Semantics
- 33:39Mesos Modules
- 34:56Linux Distributions
- 35:30Services on the same node...
- 36:14Spreading out Master Nodes
- 37:54Agent Attributes
- 39:20Cluster Upgrades
- 40:43UPGRADE PROCEDURE
- 43:42Software Upgrades
- 44:44Day 2 Operations
- 46:30METRICS
- 47:10DC/OS METRIC SOURCES
- 47:57Production Checklist
- 48:07MESOS CHECKLIST
- 49:03MESOS MASTER CHECKLIST
- 49:30ZOOKEEPER CHECKLIST
related talks
39mHow we learned to love the Data Center Operating System
01h 04mLaunching patterns for containers - it's more than just scheduling
23mHow cluster managers affect the landscape of modern distributed computing
53mDocker as the Building Block for Datacenter-Scale Applications
32mTechnical View: Comparison of Container Orchestration and Management Systems
27mThe Night of the Living Containers: Horror Stories from the DevOps World
27mKeynote Panel: Future of Cluster Management
33mDistributed Scheduler Hell