Tuning and Managing Loggregator, the Cloud Foundry Logging Subsystem
summary
Loggregator breaks into Metron agents on each cell, Dopplers as the middle tier, and traffic controllers; Metron maxes at about 8,000 msgs/s on typical VM hardware and operators should plan around 1,000 app logs/s per cell to avoid loss. Dopplers recommend staying under ~4,000 msgs/s each, provide per-sink buffers (defaults raised from 99 to 9,999) and intentionally drop whole buffer when a sink overflows. Concrete fixes: add Dopplers, balance across availability zones, increase buffer sizes, use multiple nozzles or multi-line packing, and tune syslog drain consumers.
AI-written summary · about
chapters (12)
- 0:00Introduction to Loggregator
- 1:23How Loggregator works
- 3:08App logs flow explained
- 4:31Log loss and its causes
- 7:18Tuning fundamentals
- 10:25Metron tuning tips
- 12:51Multi-line log handling
- 16:03Doppler configuration
- 22:25Traffic controller scaling
- 25:56Common Loggregator issues
- 28:48Slow consumer solutions
- 30:54Contacting the team
related talks
32mAddressing Delivery Reliability in Loggregator
30mMonitoring Cloud Foundry: Learning about the Firehose
29mCloud Foundry Logging & Metrics
31mDefining Service Level Objectives for Loggregator
21mLogging at scale using Graylog - Billion+ messages, 100K req/sec
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
33mSeeing the Forest for the Trees: An Operator’s Guide to Observability of CF with...
31mThe Long and Winding Road to Application Logging [A]