Making the Most Out of Kubernetes Audit Logs
Talkat KubeCon + CloudNativeCon North America 2019 · on Tuesday, 2019-11-19· at 10:55· Room 6C - San Diego Convention Center, Upper Level·
summary
Datadog reports running clusters of 1000+ nodes and demos show the API server emitting about 900 calls per second on a 2,500‑node cluster. Audit logging is enabled with two API server flags and an audit policy file that controls stages (RequestReceived, ResponseComplete/Started) and levels (Metadata, Request, RequestResponse). Examples: kubelets cause heavy GETs of node/configmap/secret, 403s trace to orphaned nodes and a Traefik service account listing secrets, and 422s expose garbage‑collector failures caused by mutating webhooks.
AI-written summary · about
chapters (28)
- 0:00Intro
- 2:19Control plane
- 3:42A few more GET examples
- 4:31Describe resource
- 8:04What happened?
- 8:57Why was it authorized?
- 9:46Another GET call
- 10:21Watches
- 11:08Create call
- 11:51Outline
- 14:07Gotchas
- 15:08Recommendations
- 17:05Top API users?
- 18:36Grouping by users is not helping
- 19:42Verbs on "node" for a kubelet
- 20:30Verbs on "configmaps"
- 20:54Group by resource name
- 21:53List latency by resource
- 22:53Compare cluster performances
- 23:27Takeaways
- 24:40Creating a simple deployment
- 24:56Sequence
- 27:43Node API calls
- 29:35Troubleshooting
- 30:31Status codes
- 33:08What is failing?
- 33:33What is happening?
- 34:53Conclusion
related talks
25mAuditing in Kubernetes 101
35mUsing K8s Audit Logs to Secure Your Cluster
33mWho Shot the Cluster? - Audit Logging in Kubernetes - Marian Lobur & Mik Vyatskov, Google
22mAudit in Kubernetes Now, and in the Future
20mKubernetes audit log best practices
31mAuditing in Kubernetes 101
26mUsing Go to Scale Audit Logging at Cloudflare
32mThreat Hunting at Scale: Auditing Thousands of Clusters With Falco + F...