Unconference Session: Reliable K8s Operator: How to Survive Eventual Consistency
summary
Reports of a DaemonSet creating duplicate pods every five minutes and a scheduler stuck two hours behind due to etcd gRPC watch starvation motivate concrete fixes. Kubernetes adds comparable resourceVersion and a controller stale detection pattern that delays reconciliations until a controller sees its previous write, rolled to beta in 1.36. Controller-runtime is proposing compatible guarantees; the session explores how to adopt the new API, measure cache lag, and whether the write-read blocking tradeoff prevents cascading failures at 5k+ node scale.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:06Eventual consistency matters
- 2:34Stale data consequences
- 5:21Kubernetes 1.36 changes
- 8:12Stale control detection
- 11:05Controller runtime discussion
- 12:52Kubernetes at scale fragility
- 15:46Summary
related talks
46mOperators Live in the Past: Designing Reliable Kubernetes Controllers
31mIt's all about reconciliation - anatomy of a kubernetes controller
34mWriting a Kubernetes Operator: the Hard Parts
24mKubernetes Operators with Go
35mKubernetes, Lessons Learnt
36mBuilding a Fault-Tolerant Resources Controller on Kubernetes
21mStory of a Kubernetes Go Operator
43mDeep Dive: Apps SIG