Scaling Fluentd From 0 to Millions of Events
summary
Fluentd and Fluent Bit perform best with a lightweight agent at endpoints and an aggregator model that centralizes parsing, filtering and config changes. Benchmarks show a single worker handles ~12.5k events/sec, two workers ~25k, and 4 workers -> 30k eps before CPU saturates; adding vCPUs and workers scales roughly linearly. Shorter flush intervals (3s vs 30s) smooth CPU spikes. Message size matters: 10B messages -> 63k eps, 100B -> 50k, 400B -> 30k.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:33Considerations in production
- 2:15Logging operations
- 3:17Challenges of IT Ops team
- 4:32Implementation models
- 5:50Benefits of Aggregator
- 7:20Fluent Bit and Fluentd
- 9:03Aggregator Performance?
- 11:50Characteristic: CPU usage
- 13:13flush_interval
- 14:04Performance measurement
- 15:07Scale performance : # of workers
- 16:58Scale performance : # of CPU
- 18:07Scale performance : Message size
- 19:13Conclusion
related talks
22mPetabyte Scale Logging with Fluentd and Fluent Bit: A Use Case From...
34mFluentd and Fluent Bit
13mFluent Bit
36mDeep Dive: Fluent Bit
22mLogging: Fluentd & Fluent Bit
09mWelcome to FluentCon! What's new with Fluentd & Fluent Bit
14mHow We Implemented High Throughput Logging at Spotify
39mMaking Logs Work for you with Fluentd