PIMS: Fleet-Wide Datacenter Maintenance with Minimal Capacity Buffer and Predictable Lat…
Talkat 20th USENIX Symposium on Operating Systems Design and Implementation (OSDI '26) · 2026-07-13..07-15·
summary
PIMS is a production system running five years across millions of servers and tens of thousands of services that enforces a one-domain regional contract (at most one maintenance domain down at once), aligns service and maintenance domains, and sequences domains in a maintenance train that visits every domain at least once in 45 days. It uses hardware rebalancing and a dual-buffer design, achieves a drain SLA for buffer handoff (30 minutes target; four such fleet events in 2025 resolved <20 minutes), reports a breach rate ~1 per million, and shrinks capacity buffer ~15% in Q1 2025 while meeting rollout SLOs (45 days OS, 90 days firmware).
AI-written summary · about
chapters (11)
- 0:00PIMS and fleet maintenance
- 1:45Seamless predictable maintenance
- 2:51Aligned domains in IAS contract
- 4:17Single-domain-down contract necessity
- 6:51Extending contract to service domains
- 8:59Hardware spread and rebalancing
- 9:20Buffer strategies for maintenance
- 12:09Consistent service expectations
- 13:56Maintenance sequencing guarantees
- 15:43Disaster recovery expectations
- 16:23Conclusion and benefits
related talks
15mKeeping a Hypervisor Fleet Up to Date with Minimal Customer Disruption
33mScalable methods to patch and upgrade your hybrid fleet of Oracle databases
27mService Degradation at Scale: Creating Instant Capacity
35mCapacity Planning in Four Parts: Telling the Future without a Crystal Ball
31mServers Are Not the Hardest Thing to Orchestrate
07mLightning Talk: Mission: Impossible - Building Self-Healing Disaster Recovery With...
33mHow and why we lowered our SLO
01h 02mAutomating the component maintenance - Fleet management at Spotify
