400 Billion Messages a Day with Kafka
summary
PayPal runs 400 billion messages a day across 50+ Kafka clusters, ~3,000 topics and 7 PB of disk, upgrading from 0.8 to 1.1 while supporting Java, Python and Node.js clients. Built an operational dashboard called cough cmon, a canary app, monitoring interceptors, a data-loss auditor, self-service onboarding and a configuration service plus control plane to orchestrate topic migration and mirror-maker changes. Result: onboards drop from weeks to minutes and outage recovery falls from hours/days to minutes.
AI-written summary · about
chapters (13)
- 0:00Introduction and agenda
- 1:49Kafka at PayPal today
- 3:36Early Kafka challenges
- 5:02Growth and scaling hurdles
- 6:26Operational dashboard CoughCmon
- 7:50Client-side monitoring
- 9:15Data loss auditing
- 12:13Onboarding and capacity management
- 15:14Cluster group model
- 20:12Load migration automation
- 25:32Control plane architecture
- 34:19Future roadmap
- 35:41Conclusion and Q&A
related talks
35mKafka Deployment at Scale
18m"Kafka's Metamorphosis"
05mPREVIEW: 1 Day, 1 Data Hub, 100 Billion Messages
38mKafka Message Broker Under High Load
32mScaling your Kafka streaming pipeline....
50mApache Kafka + Zookeeper = 3.5 Million Writes Per Second by Design
28mTackling Kafka, with a Small Team
30mEvent Streaming and the Future of Applications