Using Go to Scale Audit Logging at Cloudflare
summary
Cloudflare builds the audit pipeline in Go using Kafka, Kubernetes and Redis and faces a jump from ~3.2k msgs/s for one product to ~2.5–3k msgs/s systemwide while consumers only process ~450–600 msgs/s. Bottlenecks are a >5s request->audit transform, slow downstream enrichments and per-row DB inserts. Solutions: batch consumption (batch size 500, flush 5s), increase partitions from 3 to 6 and run four pods, transform in parallel with goroutines/channels, cache enrichments in Redis and use batch DB inserts — transform time falls to ~3s and DB latency drops.
AI-written summary · about
chapters (20)
- 0:00Scaling audit logs with Go
- 1:24Why audit logs matter
- 3:12Cloudflare’s Go architecture
- 4:16What are audit logs
- 6:22Audit log pipeline
- 7:03Kafka for event streaming
- 8:48Scaling audit logs challenges
- 10:36Bottlenecks with metrics profiling
- 13:07Profiling transformation delays
- 14:12Database insertion hurdle
- 15:18Kubernetes Kafka scaling
- 16:22Kafka consumer groups
- 18:55Kafka batch consumption
- 20:20Optimizing consumer groups
- 21:24Go concurrency
- 22:50Goroutines for requests
- 23:31Redis caching
- 24:15Batch DB insertion
- 24:58Key takeaways
- 25:41Q&A
related talks
36mMaking the Most Out of Kubernetes Audit Logs
21mLogging at scale using Graylog - Billion+ messages, 100K req/sec
28mManaging and Scaling Real Time Data Pipelines using Go
44mScale Your Auditing Events
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
46mBuilding a scalable logging platform
06mHow We Achieved 100x Scale
37mA Needle in a Haystack: How to Find a Threat Hidden in Over 6 Billion Logs Per Day