How We Implemented High Throughput Logging at Spotify
summary
Fluentd is CPU-bound on a single core and causes massive silent loss — Spotify estimates ~80% of logs dropped as the deployment grows to ~1,700 namespaces. The team replaces custom agents with fluentbit high-throughput exporter, creates ~2,000 per-project log sinks, and measures per-project throughput with Prometheus. Costs spike during rollout, so they implement automated alerts that trigger Pub/Sub-driven per-project sampling at 10% and expose controls in Backstage, which reduces ingestion and brings costs under control.
AI-written summary · about
chapters (7)
- 0:00Logging infrastructure history
- 2:38Identifying log bottlenecks
- 4:01Evaluating alternative designs
- 6:19Implementing the new architecture
- 8:06Managing costs and throughput
- 11:01Current limitations and fixes
- 12:40Best practices for logging
related talks
22mPetabyte Scale Logging with Fluentd and Fluent Bit: A Use Case From...
35mHigh Throughput with Low Resource Usage: A Logging Journey
31mMy Logs Cost More Than My Services
22mLogging: Fluentd & Fluent Bit
46mBuilding a scalable logging platform
22mFluent Bit - Swiss Army Tool of Observability Data Ingestion
14mIntegrating Fluent Bit Into a PaaS
13mFluent Bit