"TOGA: Checklists and Safety Measures for "Yoloing" in Prod"
summary
Covers concrete rules and failures: a real incident generates 50+ checklists, debris severs 650 wires and forces a rethink to “what’s still flying,” landing with a 100 m margin. Shows concrete mitigations: upgrade controller VMs first, staged cluster rollouts, rollback plan for driver changes, using IPMI virtual consoles to load an old kernel and an Ansible playbook to remove a bad kernel. Emphasis is on measurable thresholds, clear ownership, and safe abort vs commit decisions.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:21Speaker background
- 3:09Safe changes in production
- 4:54Aviation safety for engineering
- 7:41Auto landing in aviation
- 10:32QF32 engine failure case
- 13:38Managing incident alerts
- 18:20VM hypervisor upgrade example
- 22:09Kernel reboot failure example
- 24:57Automation vs safety nets
- 26:42Key takeaways
related talks
48mMove fast Build things… Safely!
46mPlanning for and handling failures from open hardware, aviation, to production at Google
34mMove Fast Build Things… Safely!
36mMove fast Build things… Safely!
22mFailing Over without Falling Over
50mLearning in Production (or Why the Apollo 11 Landing Nearly Failed)
44mPlanning for & Handling Failures from Open Hardware, Aviation, to Production at Google
31mManaging Unmanageable Complexity