Risky business: taking risks in production
summary
Comcast X1 ships a new personalized "For You" experience built from Spark jobs, web services and Couchbase, validated with A/B testing, rollout via canaries and feature flags, and protected by circuit breakers and anomaly detection. Capacity planning doubles allocation weekly for six weeks, limited QA with fast rollback, and occasional chaos exercises. The deployment lands without a technical rollback, but triggers product tensions between algorithmic ranks and editorial curation and produces measurable engagement changes.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 4:07AV testing
- 5:14Product risks
- 7:35Operational risks
- 9:11Quick rollback
- 10:09Canaries
- 11:07Outlier detection
- 12:05Circuit breakers
- 12:41Dynamic configuration
- 13:19Scope configuration
- 15:45Personal risk
- 18:13Personal consequences
- 19:00Repercussions
- 19:51Personal risks
- 22:59Closing thoughts
related talks
47mImproving Business Resiliency with Chaos Engineering
08mKeynote: How Comcast is Working to Transform our Product Delivery Experience
37mEmbracing Change: Infrastructure Experiment...
25mChanging the Development Culture at Comcast and Our Evolution from Legacy SOA to Cloud Native
22mScaling Culture of Resiliency in the Enterprise at Charter Communications
29mFast, Safe, and Continuous -- Picking all 3 at Netflix
34mChaos Engineering:The good, the bad & the ugly
38mChaos Engineering