Kubernetes API Server Performance Clinic: Auditing, and Priority...
summary
Shows how to diagnose API latency using Audit logs and Prometheus metric apiserver_request_duration_seconds to find noisy-neighbor controllers, then apply Kubernetes Priority and Fairness (APF) (flowSchemas + PriorityLevelConfiguration queues/seats) to protect system calls (node heartbeats, leader election) at 10k+ pod scale. Demonstrates a live outage that produces 429 errors, fixes via API streaming (WatchList), client-side caching, and tuned APF queues, cites a GitHub repo and notes APF entered core in 1.20 and was publicly enabled after 1.29.
AI-written summary · about
chapters (14)
- 0:00API server performance challenges
- 3:16API server bottleneck
- 7:50List queries overload
- 9:13Request prioritization need
- 10:58API Priority and Fairness
- 13:26APF streamlines requests
- 16:38Flow schema config
- 18:44Request queuing in APF
- 20:27APF metrics to monitor
- 21:08Demo: API server overload
- 27:52Demo results and 429 errors
- 31:44APF config best practices
- 32:46Resources and GitHub repo
- 33:08Q&A
related talks
30mAPI Priority and Fairness: Kube-APIServer Flow-control Protection
31mThe Life (or Death) of a Kubernetes API Request, 2025 Edition
30mOperating within Normal Parameters: Monitoring Kubernetes
25mAuditing in Kubernetes 101
44mPreventing Controller Sprawl From Taking Down Your Cluster - When a Scalable Pattern Stops Being Scalable
59mKubernetes - going beyond the basics
37mMonitoring Service Architecture and Health with BPF
04mKube-Liveboard: Make Your Cluster Visible