Apache Beam and Google DataFlow IRL with Scala Scio
summary
Scio is a high-level Scala API built on the Apache Beam Java SDK that lets the same pipeline run as batch or streaming and swap runners without rewriting logic. The talk covers concrete Google Cloud integrations (Pub/Sub, BigQuery, Cloud Storage, Dataflow), notes Dataflow is fully managed and can be costly, and describes Scio’s Scala-specific features: wrapping the Java SDK with Scala futures, using macros to generate coders, support for local testing, and a demo switching a pipeline source and running it on Dataflow.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:14Beam goals and model
- 3:28Pipeline concepts and sources
- 5:41Dataflow in GCP
- 7:28Scala with Scio
- 9:17Scio advanced features
- 12:00Dataflow UI demo
- 13:36Runners and testing
related talks
01h 03mBig Data Pipelines With Scala - Part 1 of 2
48mData Streaming with Apache Beam / Cloud Dataflow
38mData processing with Apache Beam
32mWhy Use Scala in a Bigdata Context?
26mA Gentle Introduction to Apache Beam
36mData pipelines engineering made simple with Scala
42mScala: The Unpredicted Lingua Franca for Data Science
44mImprove your App UX with one model processing insights from batch and streams