Don't Let Your Kubernetes Cluster Go Wild: Ensuring Etcd Reliability
Talkat KubeCon + CloudNativeCon Europe 2025 · on Friday, 2025-04-04· at 13:45· Level 1 | Hall Entrance S10 | Room C·
summary
Etcd v3.6 gains a robustness test framework that injects faults and checks strict serializability by running randomized valid workloads against a clean cluster, recording client/server logs, and validating with a state machine plus porcupine linearizability checker. Faults are injected with gofail, lazyfs and a reverse proxy; some race bugs require 100–200 runs to reproduce. The system surfaces concrete regressions (example: a revision jumping 165→164), gets reproduced reliably, and is added to CI and regular biweekly runs.
AI-written summary · about
chapters (14)
- 0:00etcd reliability intro
- 1:49Strict serializability in etcd
- 4:19Handling concurrent requests
- 6:44Real-world etcd failures
- 7:46Why robustness testing
- 8:51Robustness vs other tests
- 10:39Robustness test design goals
- 12:07Robustness test execution stages
- 14:14Failure injection tools
- 16:25Collecting outputs
- 17:49State machine validation
- 20:00Revision regression bug demo
- 22:29Future work improvements
- 23:19Q&A on reproducibility
related talks
31mDemystifying Kubernetes Failure Scenarios Caused by etcd
43mSelf-hosted, Scale, and Federation with Kubernetes v1.4 and Beyond
27mRunning Kubernetes in Production A Million Ways to Crash Your Cluster
46mContinuous Kubernetes Security
35mLessons Learned From Etcd the Data Inconsistency Issues
19mDr etcd; or; How I Learned to Stop Worrying and Love the Datastore - Nick Young, VMware
20mLessons Learned from Operating ETCD
42metcd: distributed locking and service discovery