'No One at Google Uses MapReduce Anymore' - Cloud Dataflow

on YouTube ↗47m

" summary

Explains Google's move beyond MapReduce to newer models—Flume for pipeline definition and MillWheel for real-time dataflow—and introduces Cloud Dataflow, a public tool for specifying and running both batch and streaming pipelines at Google scale. Covers the post-MapReduce dataflow model, the underlying flow optimization algorithms, and how Cloud Dataflow automatically deploys and manages many machines for parallel processing.

AI-written summary · about

" chapters (14)

1. 0:00Cloud Dataflow intro
2. 1:07MapReduce limitations
3. 3:39MapReduce to Dataflow
4. 8:36Associative reductions
5. 11:49Flume Java primitives
6. 20:10Execution graph optimization
7. 25:08Graph pattern matching
8. 31:29Optimizing execution
9. 33:15Dataflow vs others
10. 36:41Real-time streaming
11. 38:31Q&A Java Pig
12. 41:18Q&A Local execution
13. 43:02Q&A Flume Java
14. 44:33Q&A Deployment autoscaling

" related talks

• 41mReal-time Data Processing with Cloud DataflowFernando Ultremare · GDG Campinas Data Fest 2017
• 24mApache beam and google cloud dataflowMoshe Shamy · PyCon Israel 2017
• 41mGoogle Dataflow: New open model for batch & stream processingFelipe Hoffa · Berlin Buzzwords 2016
• 39mApache Beam and Google Cloud DataFlowMete Atamel · GDG DevFest Ukraine 2017
• 26mBeam and dataflow: a perfect combination of parallel processingEl Mehdi El Khayati · BlaBlaConf 2022
• 26mETL pipeline with GCP Dataflow​Oleksandr Reshetnik · Devoxx Ukraine 2021
• 38mShall We Talk About Data Pipelines?Cesar Nogueira · GDG DataFest 2018
• 44mBig Data Science on Google CloudFernando Ultremare · GDG DataFest 2018