DBTB INT Eugene Kirpichov r

on YouTube ↗04m

" summary

A Google Cloud Dataflow team builds and ships a production feature called No Shard Left Behind, a dynamic worker balancing system for Dataflow that took about three years to get right and now has a public blog describing how it works. The work sits inside the open-source Apache Beam effort, which unifies batch and streaming processing, runs over multiple platforms, and invites contributors to use its SDKs and runners.

AI-written summary · about

" related talks

• 43mNo shard left behind: Dynamic work rebalancing in Apache BeamMalo Denielou · Flink Forward Berlin 2016
• 26mBeam and dataflow: a perfect combination of parallel processingEl Mehdi El Khayati · BlaBlaConf 2022
• 24mApache beam and google cloud dataflowMoshe Shamy · PyCon Israel 2017
• 46mBuilding an Apache Beam streaming runtime in GCPSergei Sokolenko, Reuven Lax · Flink Forward Europe 2019
• 39mApache Beam and Google Cloud DataFlowMete Atamel · GDG DevFest Ukraine 2017
• 26mETL pipeline with GCP Dataflow​Oleksandr Reshetnik · Devoxx Ukraine 2021
• 10mDemocratizing the pipeline with Apache Beam and DataflowTyler Akidau, Reuven Lax · Strata Data Conference 2017 - New York, New York
• 41mGoogle Dataflow: New open model for batch & stream processingFelipe Hoffa · Berlin Buzzwords 2016