Ironic Operations at CERN: Our top 5 challenges and how we handle(d) them
summary
CERN runs Ironic since 2017 and manages almost 9,000 physical nodes; five concrete fixes are described: enrolling live servers by using fake drivers to pass cleaning, mapping physical machines by adding nodes one-by-one to placement aggregates via a script, giving projects a max_instances property and using Ironic multi-tenancy for visibility, repairing missing inventory with an active introspection container instead of reinstalling IPA, shortening hardware-manager updates by configuring IPA-builder to pull hardware manager at boot, and cutting resource-tracker time from ~3 hours (at ~5,000 nodes) to ~15 minutes by splitting into 20 conductor groups plus a leading group with five controllers and fast-track; also a collected “GRUBsetta” of boot errors helps diagnostics.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:06Transparent node adoption
- 2:54Mapping nodes to Ironic
- 4:19User-facing resource overview
- 5:40Missing inventory data
- 7:07Hardware management cycle
- 8:31Scaling issues and fixes
- 10:36Common GRUB errors guide
- 11:19Q&A session
related talks
13mProviding Continuous Integration at CERN
20mCERN's 1500 Drupal Websites on Kubernetes: Sa...
50mOps issues
30mThe Bitter and the Sweet of Running a Planet-Scale Build &...
29mTowards Sustainable Multi-Cluster Management in Real-World Academic Infrastructure
29mA Multi-Cluster, Multi-Cloud Infrastructure with GitOps at CERN
27mHandling 100000+ visitors at the world's largest physics lab (CERN)
15mPuppet and containers: A multi-tenancy deployment