Scaling Log Collection with Fluent Bit
Talkat FluentCon: Cloud Native Logging Day with Fluent Bit and Fluentd Europe 2021 · on Tuesday, 2021-05-04·
summary
Migrating Fluentd to Fluent Bit and then enabling multi‑workers raises throughput: CloudWatch pipelines go from ~8,500 logs/s to significantly higher with workers enabled, Kinesis/firehose examples show ~4,000 logs/s improving similarly. Benchmarking shows CPU rises slightly while memory stays stable. Recommendations: disable heavy k8s label/annotation enrichment, enable filesystem buffering and 50–100MB memory_buffer_limit, prefer the cloudwatch_logs plugin configuration provided. Concurrency support varies: workers enable dedicated threads, cloudwatch_logs lacks concurrency (single worker only); destination throttling and API ordering remain scale limits.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:42Focus Areas
- 2:07Filter Characteristics
- 5:09Best Practices
- 8:45Coroutines
- 11:35Workers
- 12:47Concurrency Workers
- 16:56Concurrency Support
- 17:54Example Outputs
- 20:04Conclusion
related talks
27mUnder the Hood with Fluent Bit Operator: Kubernetes-native Log Processor
34mPower of O11y with FLUENT BIT in AWS EKS environment
19mA New Approach to Logging as a Stack: Fluent Bit + PostgreSQL (FPS)
35mIntro: Fluentd
34mFluentd: A Complete Logging Ecosystem for Kubernetes
05mLightning Talk: What I Wish I’d Known about Fluentd with Kubernetes - Bryan Boreham, Weaveworks
57mTelemetry Showdown: Fluent Bit vs. OpenTelemetry Collector - A Comprehensive Benchmark Analysis
31mLoad Testing Kubernetes: How to Optimize Your Cluster Resource Allocation in Production [I]