Batching vs. Streaming - Scale & Process Millions of Measurements a Second
summary
Starts with a Lambda/Kinesis/DynamoDB/S3 pipeline handling ~5,000 measurements/sec, scales to ~75k/sec, then moves to Spark on EMR for batch (255k/sec) — reliable but 3-hour jobs and huge 7 PB storage. Switching to Apache Flink yields sustained streaming at ~12 million measurements/sec, cuts storage to ~2 PB and reprocesses 67 trillion measurements but suffers state/backpressure, operational complexity and cost. By 2020 a hybrid ECS stream-batch approach pre-rasterizes common tiles, drops API latency to ~1s, and accepts dual ingestion engines and higher compute/storage tradeoffs.
AI-written summary · about
chapters (7)
- 0:00Intro
- 3:31Streaming machine state data (2014)
- 5:39Streaming agronomic data (2016)
- 10:10Batch agronomic Data (2017)
- 12:21Streaming agronomic data (2018)
- 16:27Streaming agronomic data (2020)
- 18:47Beyond 2020
related talks
39mDeploying Batch and Streaming Architectures on AWS
38mHow to handle 100k events per second
29mData Acquisition: Overcoming Challenges
40mThe Seven Deadly Sins of Streaming
42mBuilding a Streaming (E-health) Data Pipeline: When and How?
39mEvaluating Streaming Framework Performance for a Large-Scale...
36mStream All The Things!
01h 01mDealing With Data Panel