Fluent Bit - Swiss Army Tool of Observability Data Ingestion
Talkat FluentCon: Cloud Native Logging Day with Fluent Bit and Fluentd Europe 2021 · on Tuesday, 2021-05-04·
summary
Built a Fluent Bit pipeline that ingests syslog/tcp/tail inputs in AWS Fargate, cleans nested JSON with nested JSON cleanup Lua scripts and regex parsers, and forwards raw logs to Loki while emitting log‑derived counters to a Prometheus Pushgateway. A custom Golang output plugin uses the Fluent Bit SDK to define counter metrics (job=cloudwatch_logs), push them fire‑and‑forget to Pushgateway, and let Prometheus scrape them; solution runs at scale, lowers AWS cost, and shows working counters and queryable rate charts in Grafana.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 0:36Shoutout
- 1:33What is it
- 2:10It is fast
- 2:27Its container friendly
- 2:56Cloud provider adoption
- 4:01Check the boxes
- 4:20Create
- 4:49Plugin Architecture
- 6:23Plugin Configuration
- 8:19Input
- 9:04Forward
- 10:23Parser
- 11:53Flexibility
- 12:34Troubleshooting
- 14:52Pipeline
- 21:27Lessons Learned
related talks
22mLogging: Fluentd & Fluent Bit
14mFluent Bit: Past, Present, and Future
33mFluent Bit V2.0: Unifying Open Standards For Logs, Metrics & Traces
36mDeep Dive: Fluent Bit
34mFluentd and Fluent Bit
14mIntegrating Fluent Bit Into a PaaS
28mBuilding Observability Pipeline with Fluent-Bit
34mPower of O11y with FLUENT BIT in AWS EKS environment