How Preemption Works in Nomad
summary
Nomad adds a scheduler preemption feature that makes room for higher-priority work by evicting lower-priority allocations instead of waiting for autoscaling; priorities range priority 0–100 and the scheduler uses bin‑packing plus an iterative per-priority selection to avoid 2^n subset enumeration. It scores candidates by a Euclidean distance metric across resources, filters supersets to avoid waste, and scores across nodes for the best place to evict. Exposed fields preempted_allocs/preempted_by and nomad plan integration provide visibility; preemption shipped in Nomad 0.9 beta (system jobs) with enterprise support for batch/service coming.
AI-written summary · about
chapters (20)
- 0:00Introduction to Nomad
- 4:30Interacting with Nomad
- 5:27Scheduler mechanics
- 8:03Priority inversion problem
- 8:22Example of resource pressure
- 10:59Understanding preemption
- 11:49Implementation considerations
- 12:29Intra-node allocation logic
- 15:59Subset selection complexity
- 16:21Iterative optimization
- 16:56Distance and scoring functions
- 17:45Preemption algorithm details
- 22:39Summary of selection process
- 23:17Internode scoring problems
- 23:42Cross-node evaluation
- 24:42Final placement logic
- 25:12Visibility and debugging
- 26:02Dry run capabilities
- 26:21Rollout and availability
- 27:14Conclusion
related talks
35mLiving the Nomadic life
12mNomad: Lightweight Orchestration That Complements Kubernetes
01h 06mSchedule AI workloads efficiently on Nomad with node pools
27mNomad Best Practices for Reliable Deploys
33mAbstracting Nomad: Patterns for Deploying to Nomad at Scale
36mHow Nomad can shoot you in the foot, and what to do to dodge it
29mExtending Nomad with Plugins
30mAutoscaling with HashiCorp Nomad