Drivetribe’s Kappa Architecture With Apache Flink®
summary
Drivetribe builds a Kappa style event-sourced pipeline that treats Kafka as the source of truth and uses Apache Flink to compute real-time, personalized feed aggregates and heavy offline counters. The system stores events in Kafka (some topics compacted every ~30 minutes, bulky topics retained 3–6 months with cold backup), runs aggregations with Flink (HyperLogLog counters, semigroup merges), serves views from Elasticsearch, and uses active/inactive mirror clusters to deploy code without user downtime. Trade-offs are eventual consistency and extra operational complexity but the approach handles billions of events and replay from cold storage.
AI-written summary · about
chapters (23)
- 0:00Introduction
- 2:29Principles
- 3:24naive approach
- 4:08queries
- 4:31try time
- 4:51right path
- 5:36cant scale
- 5:51concurrent mutations
- 6:38modelvolution
- 7:12Rethink the problem
- 8:21Event consumers
- 8:43Benefits of Kappa
- 9:50Kappa Architecture
- 11:26Tradeoffs
- 12:53Kafka
- 13:17Flink
- 13:38Elastic Search
- 14:02Putting Everything Together
- 16:51Issues
- 17:19Active vs Inactive
- 18:16Wrong button
- 19:13Longrunning jobs
- 29:10Realtime counters
related talks
30mBuild and run streaming applications with Apache Flink and Amazon Kinesis Data Analytics
41mBeyond the Hype: When Apache Flink Solves Real Problems
40mLarge-scale near-real-time (NRT) data analytics platform empowered by Apache Flink
45mApache Flink — Real-Time Data Processing
43mA Data Streaming Architecture with Apache Flink #bbuzz
38mApache Flink: A Very Quick Guide
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink
40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobs