Fluent Bit Open Source Project & AWS
summary
Fluent Bit is a lightweight C-based log collector that AWS integrates via regional ECR images and plugins for CloudWatch, Kinesis Firehose and Streams, demonstrated delivering logs to S3 then queried with Athena; tests show order-of-magnitude lower resource use than Fluentd for equivalent pipelines. The project reports about 300,000 daily deployments, runs as a DaemonSet to gather Kubernetes logs with metadata, now adds an on-edge SQL-like stream processor for millisecond filtering/aggregation, and implements config validation to prevent bad configs.
AI-written summary · about
chapters (6)
- 0:00Start
- 7:54Logging Cholenges
- 14:07Logging & Routing
- 18:19Why did we choose Fluent Bit?
- 19:45Regional Amazon ECR Repositories
- 22:05What should we work on?
related talks
31mFluent Bit & OpenSearch: The Power Couple of Open Source Observability
36mFluentd Project Intro – Eduardo Silva & Masahiro Nakagawa, Treasure Data
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...
39mKeynote: CNCF Project Updates
06mIgnite: A day in the life of an OSS maintainer
38mFrom Issue to PR Merged: A Fluentd “Tail”
23mAWS and open source: The cloud and open source are better together
38mBuilding Zero-Instrumentation Observability