Logs Collection With eBPF: Like a Super-Powered Postal Service
summary
Claims a replacement for file-based collectors using eBPF probes on write/writev syscalls to capture logs into a kernel ring buffer batching and send them to a user-space collector. Benchmarks on EKS with 100–10,000 logs/sec compare Promtail and Fluent Bit to an eBPF sensor (Flora), showing roughly 40x CPU reduction per node. Implementation removes file IO, enables cross-container batching and kernel-side filtering, but ring buffer size limits durability so files remain useful for cold storage.
AI-written summary · about
chapters (9)
- 0:00Introduction and motivation
- 0:25Traditional log collection
- 4:13Understanding log mechanisms
- 7:33Inefficiencies in current tools
- 11:08The eBPF solution
- 15:43Optimizing log pipelines
- 18:26Performance benchmarks
- 21:13Future potential and wrap-up
- 24:07Q&A session
related talks
22mPetabyte Scale Logging with Fluentd and Fluent Bit: A Use Case From...
35mAn Exemplary Path: Leveraging EBPFs and OpenTelemetry To Auto-instrument for Exemplars
22mLogging: Fluentd & Fluent Bit
37meBPF: Revolutionizing Observability and Telemetry for Modern Applications
42mCommunity Driven Breakout Session: eBPF and Auto-Instrumentation
40mLive, Laugh, Log
27mStep By Step Kubernetes Observability with eBPF
25mMaking Observability Work at the Edge