Webhook Topology and Admission Latency: Lessons from Migration
summary
Migrating Gatekeeper policies to Kyverno’s ValidatingPolicy yields a surprising production effect: per-policy webhook entries cause the API server to rebuild cached CEL programs when the validatingwebhookconfiguration’s generation increments, producing P99 from <1s to 5–6s while the policy logic itself runs at ~400µs. Observations use kubectl watch on the validating webhook config, note that the webhook controller runs every 10s and nondeterministic ordering of policies causes spurious updates; deterministically sorting policies before diffing prevents the rewrite and removes the latency spikes.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 1:02Migrating policies Gatekeeper to Kyverno
- 2:07Unexpected latency spike
- 3:51Investigating and mitigating
- 5:13Root cause of latency
- 9:03Lessons learned
- 10:31Q&A
related talks
35mAdmission Webhooks: Configuration and Debugging Best Practices
35mShift-Left Policy Validation in Kubernetes With Kyvern...
35mEnforcing Bespoke Policies in Kubernetes [I]
29mKyverno Chronicles: A DevSecOps Tale
29mEVAPorating Kubernetes Security Risk: Adopting Validating Admission Policy at Scale
24mKyverno Mutating Policies: Shifting Platform Logic Into the Control Plane
20mUsing, and mis-using K8s Dynamic Admission Control
25mCreating and Debugging Kubernetes Admission Webhooks