Patterns for building Highly Available Cluster Schedulers
summary
Analyzes concrete scheduler designs and failure modes, comparing event-driven vs level-triggered reconciliation and showing that level loops (example: 300 ms) trade simplicity for latency while Nomad’s event-driven allocator reaches roughly Nomad schedules ~1M containers <5 min but must handle lost events. Explains resource managers are strongly consistent—etcd/Paxos/Raft—and lists operational fixes: disable kernel OOM and kill in user space, canary system updates (1% rollout), account for ZFS ARC memory leaks, enforce quotas, and plan for a 45-minute region recovery budget.
AI-written summary · about
chapters (8)
- 0:00Introduction to cluster schedulers
- 3:59Why cluster schedulers evolved
- 10:27Core components of schedulers
- 16:36Handling node-level failures
- 24:19Cluster-level efficiency concerns
- 26:50Managing cluster-level failures
- 32:03Planning for data center outages
- 33:40Q&A and closing
related talks
01h 03mScaling Web Operations with Distributed Cluster Schedulers
40mNomad: Scheduling Applications at Scale
33mDistributed Scheduler Hell
37mAn Introduction to Schedulers
46mWriting Stateful Frameworks
47mNHD - A Topology-Aware Scheduler for K8s for Low-Latency & HPC Applications
39mPractical Container Scheduling: Juggling Optimizations, Guarantees, and Trade-Offs at Netflix
29mScale Your Batch / Big Data / AI Workloads Beyond the K8s...