How We Implemented a Near Real-Time Logging Solution Using Fluent Bit And OpenSearch
summary
Implemented a near real‑time logging cache on Kubernetes using Fluent Bit (daemonset, tail + OpenSearch plugin) and OpenSearch, publishing logs to a new hourly index and auto‑deleting indexes older than 80 minutes. Metadata comes from the node kubelet to avoid API overload, and Fluent Bit enforces per‑project throttling. Deployed across 15+ clusters (largest ~55 nodes), logs remain inside the data plane for compliance, and the console shows entries within 2–3 seconds.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 4:03Need for near real-time logging
- 7:40Goals and evaluation
- 9:27Fluent Bit and OpenSearch architecture
- 13:22Results and benefits
- 15:55Live demo
- 19:39Q&A
related talks
21mKubernetes Under the Magnifying Glass: Observability With OpenSearch
31mFluent Bit & OpenSearch: The Power Couple of Open Source Observability
34mGetting Started With Logging in Kubernetes
26mKubernetes and Logging: Do It Right
46mOperational Logs and Event Management for HPC S...
40mTaming Spiky Log Volumes: Maintaining Real-Time Log Accessibility with Kaldb
35mFrom Chaos To Clarity: Centralizing Observability With OpenSearch at Scale
46mBuilding a scalable logging platform