Don't Write Controllers Like Charlie Don't Does: Avoiding Common Kubernetes Controller Mistakes
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 14:30· Level 1 | Hall Entrance S10 | Room B·
summary
Shows that homegrown clients that call get/list on every reconcile burn API server cycles and that uncontrolled status writes create hundreds of status updates and startup thundering herd problems. Recommends using controller-runtime or alternatives KRT and stateDB to get cached informers, emit metrics, and avoid races. Advises patching or diff-checking before updates, reconciling the correct parent resource (gateway not HTTPRoute), and rebuilding state on each reconcile; real systems with hundreds of HTTPRoutes proved the fixes work.
AI-written summary · about
chapters (14)
- 0:00Controller writing challenges
- 1:03Charlie Don’t anti-hero
- 3:33Overusing API server calls
- 6:45Hand-rolled client pitfalls
- 8:51Controller development frameworks
- 12:23Controller runtime patterns
- 15:33Reconcile mistakes and lessons
- 19:24Best practices for controllers
- 21:13Q&A: Reconciliation reasoning
- 23:17Q&A: Multi-threaded operators
- 25:24Q&A: External system states
- 27:50Q&A: Operator testing
- 29:57Q&A: Object state comparisons
- 31:24Closing remarks
related talks
31mIt's all about reconciliation - anatomy of a kubernetes controller
24mWriting kube controllers for everyone
46mOperators Live in the Past: Designing Reliable Kubernetes Controllers
44mPreventing Controller Sprawl From Taking Down Your Cluster - When a Scalable Pattern Stops Being Scalable
36mUnderstanding Kubernetes Through Real-World Phenomena and Analogies
43mDeep Dive: Apps SIG
16mUnconference Session: Reliable K8s Operator: How to Survive Eventual Consistency
36mWriting controllers in rust