Real-time Stream Analytics and Scoring Using Apache Flink, Druid & Cassandra
summary
Platform processes about 100,000 events per second at peak and makes data available at <1 second end-to-query latency. Pipeline uses Kafka → Apache Flink (low-level DataStream API, custom ProcessFunctions, broadcasted state) → outputs to Apache Druid for analytics, Cassandra for latest scores and Hadoop for history. Built a runtime-configurable feature-engineering framework, migrated batch Spark Python jobs to Flink for a big performance boost, serves models either embedded or via async external serving, and uses safepoints and reprocessing to handle stateful job evolution.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:25Example use cases
- 2:47Architecture Overview
- 4:07Solution powered by Apache Flink
- 4:54Challenge: Feature Engineering for Anyone
- 7:04Simple Count Example
- 8:04Example Generic Features
- 8:44Challenge: Dynamic pipeline updates
- 12:16Challenge: ML Model Serving
- 15:18Challenge: Evolution
- 18:13Challenge: Maintenance
related talks
15mData science at scale using Apache Flink: Dynamic model serving and real-time feature generation
20mUsing Flink for Versatile Feature Engineering Pipelines
25mSharing what we love: The Apache Flink story at Pinterest
25mData: Moving with the flow
15mFeature Stores: Powering Machine Learning Infrastructure with Apache Flink
37mBig Data Analytics, Machine Learning, Real Time Processing
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
27mArchitecting Sub-Second Analytics Pipelines