How Uber Audits Real time Infrastructure of Trillion+ Messages
summary
Audit system Chaperone counts messages end-to-end through Kafka at Uber scale, claiming visibility over trillion+ messages per day by embedding creation timestamps and aggregating counts into timestamped 10‑minute buckets. Architecture uses a rest proxy + custom client libs that batch and ack quickly, emits audit events, and originally used a Helix-based collector consuming audit topics. Redesigned to write audit records through a stateless web service into Cassandra, replace Helix with ZooKeeper locking, reduce deployments to one per DC, and scale from ~250B to over 1T Kafka events (≈5T audit events).
AI-written summary · about
chapters (26)
- 0:00Intro
- 0:38Agenda
- 1:25Use Cases
- 2:55Uber Eats
- 4:13Fraud Detection
- 4:50Realtime Processing
- 5:36Trillion Messages
- 6:41What Kind of Data
- 7:22Uber Ecosystem
- 8:23SLA
- 14:29Kafka Life Cycle
- 16:46Why Audit
- 18:42Counting Messages
- 23:08Ordered Messaging
- 24:23chaperone service
- 25:27query results
- 26:02metrics
- 26:56latency
- 27:18Chaperon
- 30:09Solution
- 32:22Zookeeper
- 34:02Cassandra
- 35:44Comparison
- 36:48Summary
- 37:40Prototypes
- 38:32Questions
related talks
43mUnifying streaming and message queue with Apache Kafka
38mEvolution of Uber's Search Platform
39m400 Billion Messages a Day with Kafka
37mScaling Yelp’s Logging Pipeline with Apache Kafka #bbuzz
22mHow Uber Uses Your Phone as a Backup Data Center
38mKafka Message Broker Under High Load
30mEvent Streaming and the Future of Applications
27mObserving Failure in Reactive Systems