Scale By The Bay 2019: Sharanya Santhanam & Shiv Toolsidass
summary
Lyft builds a real-time pipeline using Apache Flink to transform events, Kafka (protobuf) to transport them, and Apache Druid to serve queries, achieving freshness under a minute and sub-second to single-second query latencies (target p95 <5s). The system handles hundreds of thousands of records per second, stores tens of terabytes in Druid segments, and runs a nightly batch job for segment compaction nightly and gap-filling. Production uses include alerting on experiments, geospatial dashboards, and real-time network-debugging; joins happen in Flink before ingestion.
AI-written summary · about
chapters (9)
- 0:00Introduction and motivation
- 3:25Why we chose Druid
- 7:23Real-time pipeline deep dive
- 10:12Custom Druid setup at Lyft
- 12:20Nightly batch pipeline benefits
- 13:46Onboarding product teams to Druid
- 14:46Three key use cases at Lyft
- 17:59Druid as time-series database
- 19:46Handling joins with Flink
related talks
40mLarge-scale near-real-time (NRT) data analytics platform empowered by Apache Flink
28mBest of Stream + Batch: When Streaming Analytics Isn’t
27mSucceeding with Apache Druid and Clickstream Data
32mStream Processing at Lyft with Flink and Beam
43mLeverage the power and simplicity of SQL on Apache Flink
20mAnalyzing streaming data with Apache Druid
47mBring Flink to Large Scale Production
30mData Platform Infrastructure at Rivian: A Deep Dive