Streaming Was Only Half the Fix: Retrofitting a Stable gRPC API
summary
Kubernetes CRI list calls hit ResourceExhausted errors when nodes accumulate thousands of finished containers; raising gRPC size limits only delayed failures. Engineers add six streaming RPCs to CRI with an UNIMPLEMENTED fallback for mixed-version rollouts and implement them in CRI‑O; streaming prevents single huge messages but initial server code still materializes full lists so memory stays high — profiling shows almost identical allocations (only 29 allocations difference). The rollout stops node crashes but doesn’t solve growing result sets or pod‑cleanup policy.
AI-written summary · about
chapters (8)
- 0:00Introduction and context
- 2:30Kubernetes basics and CRI overview
- 7:48Root cause of the failure
- 10:40Fixing the issue with streaming RPCs
- 13:30Version skew and rollback strategies
- 16:42Why streaming didn't change memory usage
- 18:31Open issues and limitations
- 19:54Key takeaways and conclusion
