Dude, You Forgot the Feedback: How Your Open Loop Control...
summary
Production control planes routinely accept intent or fire‑and‑forget commands that require operators to go look elsewhere to verify results, causing outages: scaling that never lands because pods never become healthy and nobody watches pod startup, blue‑green rollouts that time out then downscale the live cluster, Slack alerts that nobody owns and an expiring certificate, a Datadog log redaction that removed “e” for seven seconds, a CrowdStrike global config crash, and a GitHub dropdown that flips immediately. Fixes: show the delta, validate with the same production parser, confirm large volumetric changes, gate or pause rollouts on downstream health, and page through a real pager.
AI-written summary · about
chapters (13)
- 0:00The role of control planes
- 1:12Arrows versus hoses
- 2:45Identifying failure patterns
- 4:29Feedback in intent-based systems
- 7:27Deployment safety and visibility
- 10:24The dangers of indirect alerts
- 13:37Arithmetic and absolute values
- 15:20Validation and impact visibility
- 19:19Context and destructive actions
- 22:14Configuration parsing issues
- 25:38UI design and ergonomics
- 29:00Spooky action at a distance
- 33:01Conclusion and call to action
related talks
23mNavigating in the Dark
51mHearts of Darkness: A Spring DevOps Apocalypse
58mCultivating Production Excellence
10mOptimizing incident response
41mRestarts and rollbacks don't fix everything
40mFuzzy Lines: Aligning Teams to Monitor Your Application Ecosystem
21mQuality Gates in Production: How We Turn OpenTelemetry...
34mReading 100+ Kubernetes Post-Mortems - lessons learned