Help Protect Your Data Centers with Safety Constraints
summary
Covers concrete failures (a single rack decommission command that erased the CDN and a script that decommissioned every tunneling load balancer), the safety-check service Google built called Seriously, and the API it uses: entity + intent → allow/deny string. Describes implemented constraints: rate limits, concurrency caps, policy/assert checks, monitoring-driven service health, automatic braking, per-datacenter shard by entity rollout, short-lived approval certificates and guarded overrides with expirations. Gives lessons, testing and rollout practices and notes measurable approvals/rejections but not direct prevented-disaster counts.
AI-written summary · about
chapters (7)
- 0:00Introduction to automation
- 1:26Lessons from past failures
- 4:10Building a safety framework
- 5:46Implementing safety constraints
- 8:46Types of safety constraints
- 12:26Advanced system management
- 16:00Q&A and summary
related talks
24mHow Can You Scale It If You Don't Trust It
18mScaling APIs to Billions: Extreme-Scale Reliability
51mDesigning for Failure: Build Resilient Services
46mChaos Experiments - Datacenter Stress Testing
52mPerformance & Stability Anti patterns
40mBuilding Secure and Reliable Systems at Scale
38mScientific and Safe Chaos Engineering
27mEngineering Without a Safety Net: Where It Works and Where It Hurts