Lightning Talk: Fluentd at Scale; Keys to successful Logging Syndication
Lightningat FluentCon: Cloud Native Logging Day with Fluent Bit and Fluentd Europe 2021 · on Tuesday, 2021-05-04·
summary
Zendesk runs Fluentd as a Kubernetes DaemonSet reading Docker JSON logs from hundreds of services across multiple AWS regions, routing enriched records to outputs: Datadog, Splunk, Kafka (S3 planned). Fluentd is used as Fluentd as router with minimal in-flight parsing, keeping heavy transformations in downstream systems; tuning includes shorter flush intervals and more flush threads to control buffer memory. Deployment dates back 4 years, reliability target ~3.5 nines, and a Fluent Bit migration is under consideration.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 0:56Zendesk Architecture
- 3:00Challenges
- 4:34Why Fluentd
- 6:20Simple Business Logic
- 8:08Performance Tuning
related talks
05mLightning Talk: Flexible Logging Pipelines with Fluentd and Kubernetes
34mFluentd: A Complete Logging Ecosystem for Kubernetes
05mLightning Talk: What I Wish I’d Known about Fluentd with Kubernetes - Bryan Boreham, Weaveworks
46mBuilding a scalable logging platform
31mLogging at Scale, Billion+ messages, 100K msg/sec
42mLogging for Containers
22mPetabyte Scale Logging with Fluentd and Fluent Bit: A Use Case From...
27mUnder the Hood with Fluent Bit Operator: Kubernetes-native Log Processor