How to sleep soundly when your code is running 24/7 on millions of devices
summary
United Cloud runs EON TV on-premises serving 250,000 HTTP requests/sec to millions of devices and uses Nginx and HAProxy at the edge. A partial Apache Ignite upgrade causes an elusive Apache Ignite memory leak that escapes standard tests until developers replay Tomcat logs and mirror traffic to reproduce it. The stack adopts Canary header X UCP routing for gradual rollouts, migrates app metrics to Prometheus and an alerting pipeline with style checks, speeds CI by splitting tests and running parallel DB containers, and moves toward Kubernetes on the edge.
AI-written summary · about
chapters (14)
- 0:00Introduction and speakers
- 1:15About United Cloud
- 3:40Why on-premises infrastructure
- 9:16Sleep issues and past outages
- 13:42Performance testing with real traffic
- 15:41Canary deployments
- 17:24Monitoring and alerting improvements
- 20:12Self-service infrastructure
- 21:24Merge hell and trunk-based development
- 24:35Optimizing CI/CD pipelines
- 27:58Feature flags and code reviews
- 29:33Testing and architectural improvements
- 31:03Kubernetes on the edge plans
- 32:16Q&A and closing
related talks
24mSleeping soundly even when scaling your app to 200k request/seconds
42mPlanning To Fail
05mIgnite - What I Learned From a Dress an On-Call Nightmare
18mScaling APIs to Billions: Extreme-Scale Reliability
56mGetting all the 99.99(9) you always wanted
26mHow We Prepped Our Infrastructure for a Super-Bowl Sized Traffic
38mHighload — An Unexpected Journey
30mThe Bitter and the Sweet of Running a Planet-Scale Build &...