Batch and Streaming Processing in the World of Data Engineering and Data Science
summary
Compares concrete tradeoffs: batch gives higher control and easier correctness, streaming gives lower latency and faster model adaptation; presents a demo that deploys PySpark Logistic Regression in Spark Streaming, reads from Kafka and writes to Redis, saves models to file and updates them online via SGD, with online updates steadily improving a naive model’s accuracy; warns about feature extraction latency, unordered and duplicate events, and the operational cost of achieving exactly once delivery.
AI-written summary · about
chapters (31)
- 0:00Batch vs streaming intro
- 1:03Batching vs streaming analogy
- 2:10Phishing detection modeling
- 3:35Real-time prediction demo
- 4:16Batch vs online model updates
- 5:23Online model accuracy demo
- 6:49Spark Streaming deployment
- 7:33Online learning pros and cons
- 8:15Streaming feature extraction
- 9:22Data ingestion challenges
- 10:06Minimal streaming pipeline
- 11:11Kafka as message buffer
- 12:37Handling unordered streams
- 13:18Event vs processing time
- 14:01Time-agnostic streaming
- 15:05Approximate aggregations
- 15:48Event time importance
- 16:51Deduplication with event time
- 17:13Message delivery guarantees
- 19:03Exactly-once delivery issues
- 20:10Streaming infrastructure
- 20:51Batch processing benefits
- 21:36Latency vs correctness
- 22:19Monitoring model accuracy
- 22:40Streaming pipeline maintenance
- 23:22Engineers vs scientists bridge
- 24:47Streaming resources
- 25:08Q&A on correctness
- 25:58Q&A on online training
- 27:28Q&A Lambda vs Kappa
- 28:54Q&A raw data streaming
related talks
37mBig Data Analytics, Machine Learning, Real Time Processing
42mBuilding a Streaming (E-health) Data Pipeline: When and How?
44mImprove your App UX with one model processing insights from batch and streams
01h 21mStructured Streaming with PySpark and Azure Databricks
41mFast Data - an Overview of the Concepts, Architecture and Technology of Streaming Analytics
29mUnified batch and stream processing in python
30mSmart Streams: A Real-time framework for scoring Big and Fast Data
41mStreams to the River: Powering an Online XGBoost Classifier with Kafka