Lightning Talk: Deep Dive: How Fluent Bit Collects File Logs

on YouTube ↗10m

" summary

Fluent Bit’s tail plugin describes exact mechanics: it globs paths, calls stat/open, and tracks files by device+inode lookup (not filename), optionally persisting state to SQLite. Change detection uses iNotify on Linux or 250 ms polling (plus a 2.5 s rotation check). Reads append into per-file buffers grown by chunk size up to a max, lseeks to update offsets, parses lines into MessagePack (or JSON/multiline), and handles rotations by detecting truncation or moved-inodes with a configurable rotate-wait (≈60s).

AI-written summary · about

" related talks

• 36mDeep Dive: Fluent BitEduardo Silva, Masahiro Nakagawa · KubeCon + CloudNativeCon China 2018
• 22mLogging: Fluentd & Fluent BitEduardo Silva, Masahiro Nakagawa · KubeCon + CloudNativeCon North America 2020 Virtual
• 40mLive, Laugh, LogPaige Cruz · SREcon25 Americas
• 12mLightning Talk: Achieving Unified Observability for Cloud and Edge with FluentBitBenjamin Huo · Open Observability Day North America 2022
• 34mFluentd and Fluent BitEduardo Silva · KubeCon + CloudNativeCon North America 2021
• 21mWhy Do My Logs Suck?! A Practical Guide to Improving Your LogsPatrick Stephens · FluentCon North America 2021
• 40mFluentd Deep DiveEduardo Silva, Masahiro Nakagawa · KubeCon + CloudNativeCon Europe 2018
• 34mFluentd: A Complete Logging Ecosystem for KubernetesMasahiro Nakagawa, Yuta Iwama · KubeCon + CloudNativeCon China 2019