Learning in Production (or Why the Apollo 11 Landing Nearly Failed)
summary
The talk lays out concrete practices and historical failures: the Apollo guidance computer runs on 72 kilobytes and fires a 1202 alarm that earlier error‑handling survives, Apollo 13’s improvised CO2 scrubber actually works, and SpaceX reuses launch audio in simulations. Concrete prescriptions: small frequent releases (research shows elites deploy 200 times more), blue/green and canary rollouts, Terraform/Packer automation, ubiquitous telemetry and tracing, chaos engineering with game days and chaos monkey, and blameless incident analysis to turn incidents into learning.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 5:35Story Time
- 12:12Takeaways
- 13:14Learning from Space
- 19:47Being Prepared to Adapt
- 22:56Deployments
- 27:07Observability Operability
- 31:35Expecting Failure
- 37:04Incident Analysis
- 37:27Incidents are a Fact of Life
- 38:02Incident Analysis is Not About the Incident
- 38:19Root Cause Analysis
- 39:32What is Incident Analysis
- 40:00Blameless Postmortem
- 41:36Be Open and Honest
- 42:34Accountability
- 43:26Questions
- 43:55Collaboration
- 44:11Adaptability
- 45:13Summary
- 45:33Prepare for everything
- 46:07Questions Answers
related talks
01h 08mSmall Steps, Giant Leaps: Engineering Lessons from Apollo
46mPlanning for and handling failures from open hardware, aviation, to production at Google
57mClosing Keynote
44mPlanning for & Handling Failures from Open Hardware, Aviation, to Production at Google
30mSoftware Development Lessons from the Apollo Program
01h 06mCongratulations, you Failed!
29mLooking under the rug: the art of learning from failure
48mHarnessing Chaos; the hidden ingredient behind building better systems