Scaling Yelp’s Logging Pipeline with Apache Kafka #bbuzz
summary
Legacy scribe aggregators get replaced by Kafka as the single source of truth, with a scribe-to-Kafka connector called secretary buffering producers on hosts; topics use replication factor 3 across AZs and retention so logs can be reprocessed. Consumer parallelism grows by adding partitions and an auto partitioning service that creates partitions on spikes (example thresholds ≈700 msg/s). Production footprint: ~18 clusters, ~100 brokers, ~5 TB/day (~25 billion messages/day); monitoring focuses on consumer lag and under‑replicated/offline partitions.
AI-written summary · about
chapters (12)
- 0:00Introduction and Yelp overview
- 1:50Problem with logging pipeline
- 6:09Scribe limitations and issues
- 9:40Stat Monster metrics pipeline
- 12:31Why Kafka for scaling
- 15:17Kafka integration in pipeline
- 17:27Full Kafka-based logging ecosystem
- 19:33Kafka configuration deep dive
- 23:50Autoscaling Kafka partitions
- 25:35Kafka tooling and monitoring
- 29:50Yelp’s Kafka deployment stats
- 30:32Q&A session
related talks
38mData Pipeline with Apache Kafka
01h 00mPutting Apache Kafka to Use
50mApache Kafka + Zookeeper = 3.5 Million Writes Per Second by Design
38mKafka Message Broker Under High Load
01mScale By the Bay 2021 intro video by Opher Dubrovsky
31mLogging at Scale, Billion+ messages, 100K msg/sec
50mIntroduction to Apache Kafka
41mKafka as a Platform: The Ecosystem from the Ground Up