A Victim of Our Own Success – Overcoming the Challenges of 30,000 AIs
summary
Comcast scales from ~13,000 to ~40,000 AIs in a year and hits repeated 40% cluster CPU spikes that cause health‑check timeouts and outages; response is to decompose big sites into many smaller sites, separate dev and prod, enforce KPIs like 125% CPU hard limit and an AIs‑per‑core quota, encourage app‑level logging after a 64‑VM Loggregator cost, rebuild environments and reduced AIs by 2000, and expose metrics plus a 1,500‑user Slack and a GS LB as a service in the marketplace.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:12About APA
- 2:00Cloud Foundry
- 3:01Marketing and Training
- 4:15Growth
- 4:53Growth in 2018
- 5:30Incidents
- 6:04Impacts
- 6:46Forecasting is Hard
- 9:03Design Choice
- 11:27Meaningful KPIs
- 14:30Removing efficiencies
- 16:25Separating dev and prod
- 17:17Reducing AI footprint
- 18:30Trust in partnership
- 21:03Summary
- 22:02Open Source Platform
- 22:30Conclusion
- 23:00Questions
related talks
27mAgentic Cyber Resilience: Defending APIs Against AI-Powered Attacks
44mIntroduction to application scaling
37mExploratory Load-Testing with AI Tooling - it's Time to Ramp Up!
26mHow We Prepped Our Infrastructure for a Super-Bowl Sized Traffic
29mIntro + DeepDive: SIG Scalability
31mUse Load Testing to Build a Proper Mental Model of Your Service
39mTop Performance Challenges in Distributed Architectures
30mAbove the Clouds: Investigating Systems from 32,000 ft.