Fast Data - an Overview of the Concepts, Architecture and Technology of Streaming Analytics
summary
Explains concrete design: use Kafka as the event bus, Flink or Spark for continuous jobs, start worrying at hundreds of events/sec and switch to distributed clusters by thousands/sec; implement event time semantics, session windows with a 120s gap, watermarks (example 100ms) and allowed lateness (2s) to bound correctness; export models to PMML/ONNX and evaluate them in-process rather than via REST (API calls are too slow for per-event scoring); shows code snippets that run on JVM clusters and a working pipeline that produces realtime recommendations.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:30About me
- 1:02Big Data Revolution
- 1:52Use Cases
- 2:34Machine Learning
- 3:53Recommendation Algorithm
- 7:32Streaming Analytics Frameworks
- 7:58Deep Dive
- 9:39Flink Job
- 11:47Event Time
- 13:08Distribution
- 18:33Considerations
- 20:21Watermarks
- 26:59Click Watermarks
- 28:51Machine Learning Models
- 31:36Python Example
- 32:18XML Example
- 33:21Summary
- 36:05Questions
related talks
01h 01mFast Data Platform for Real-Time Analytics. Architecture And Approaches.
45mDeep dive into Unbounded Data Processing Systems
07mTransforming Big Data into Real-Time Product Insights
27mStream, Store, Visualize: Fast-Track Real-Time Analytics the OSS Way
19mDemystifying Modern Data Pipeline Architecture
49mStreaming Visualisation
29mBatch and Streaming Processing in the World of Data Engineering and Data Science
05mThe Future of Data Warehousing