From Exceptional Maintenance to Automated Routine Operation:...
summary
A single core datacenter migration becomes routine by replacing 15 coordinated code commits and 45 minutes of read‑only downtime with an etcd state store plus a CLI/Python library that notifies services, a distributed runner called spice track for cookbooks and dry‑run testing, and file‑watchers (comd) driving gnsd/Envoy. Traffic switches now occur multiple times per year with read‑only windows reduced to under two minutes, zero deployment commits for the switchover, and active‑active reads with a short‑lived cookie to preserve immediate edit visibility.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:07Adding second data center
- 3:17First data center switch-over challenges
- 6:28Lessons from first switch-over
- 7:54Centralized state management
- 10:27Extending state management tool
- 12:36State management in MediaWiki
- 14:21Service discovery with DNS
- 19:17Automating with Spicerack
- 23:56Read-only time under 2 min
- 24:37Traffic from all data centers
- 26:43Splitting reads and writes
- 28:07Multi-DC data store challenges
- 29:36Switch-over scheduling
- 31:00Risks of over-automation
- 32:49Key lessons from improvements
- 34:33Emotional impact of success
related talks
27mBuilding Ultra-Resilient Systems in Go
24mMigrations under Production Load: How to Switch...
41mSurviving Continuous Deployment in Distributed Systems
18mEngineering Resilience at Scale; Lessons from a Distributed Systems Transformation
38mHow we Migrated (...) Without Downtime using Elixir
44mAvailability, Latency and Cost: Withstanding Outages
53mHow to migrate a mission critical railway system
35mBuilding Systems that Last