Dr etcd; or; How I Learned to Stop Worrying and Love the Datastore - Nick Young, VMware
summary
Practical war stories about running etcd: upgrade path that lets a cluster be upgraded online in under 30 minutes, using immutable instances plus attached data volumes and lifecycle protections; upgrade to etcd 3.4 learner nodes to avoid split‑brain during joins; two real failure modes — leader election storms from slow fsync/network latency and a full datastore when revisions blow past the 2.1GB size limit (one incident caused ~80,000 namespaces); watch fsync/commit/leader_changes and DB size and have a tested recovery runbook.
AI-written summary · about
chapters (9)
- 0:00Intro
- 0:55THE BASICS-ETCO
- 4:40WHAT WE SETTLED ON
- 6:49WHAT TO DO DIFFERENTLY
- 8:32LEADER ELECTION STORMS
- 10:01LEADER ELECTION STORNS - WHAT TO DO?
- 11:01FLLING THE DATABASE EASIER THAN IT SOUNDS
- 12:48FIXING SIZE EXCEEDED ERRORS OTS TRICKY!
- 14:38UPGRADES
related talks
23mThe Secret Recipe of etcd
20mLessons Learned from Operating ETCD
46mEtcd 101: The Parts They Don't Teach You
06m⚡ Lightning Talk: Back To Basics: How To Measure Etcd Performance And Not To Die Trying
32metcd V3.6.0 and etc...
38metcd in 2026: Enhancements, V3.7 Roadmap...
26mDon't Let Your Kubernetes Cluster Go Wild: Ensuring Etcd Reliability
31mDemystifying Kubernetes Failure Scenarios Caused by etcd