High Throughput with Low Resource Usage: A Logging Journey
summary
Describes concrete engineering: Fluent Bit switches to a binary message pack payload, groups records into 2 MB chunks, and uses a hybrid in-memory + disk buffer backed by memory-mapped files to avoid syscalls and OOMs. Network work focuses on TLS keepalives, fewer handshakes, and connection reuse. Result: design changes raise throughput from ~5k/s to >30k/s on a single core, demos show ~100k/s and production runs report up to ~200k/s — the approach reduces CPU and IO overhead and works in practice.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 0:42Performance vs Cost
- 1:37Fluency
- 2:12Logging
- 4:00Debug Mode
- 4:15Logging Workflow
- 5:55Data Analysis
- 10:03IO
- 10:30Data Processing
- 11:16Output
- 14:00Pipeline
- 14:18System Code
- 14:53Processing
- 16:17Routing
- 17:32Network Setup
- 19:44JSON vs Message Pack
- 21:12Data serialization
- 22:07Memory serialization
- 22:42Memory buffering
- 30:28Demo
related talks
14mHow We Implemented High Throughput Logging at Spotify
40mLive, Laugh, Log
14mFrom Overwhelmed to Optimized: How We Cut Envoy Logging Costs by 65% While Boosting Throughput 60%
36mDeep Dive: Fluent Bit
21mLogging at scale using Graylog - Billion+ messages, 100K req/sec
29mInside the Telemetry Data Plane: Constraints, Tradeoffs, and Scale
29mLogs Collection With eBPF: Like a Super-Powered Postal Service
24mUnlocking Advanced Processing Capabilities with OpenTelemetry