Using Open Source Tech to Swap Out Components of Your Data Pipeline
summary
A Raspberry Pi cron job samples a public bike-availability API every 15 minutes and the JSON gets parsed with Apache Beam Java code so the same pipeline runs locally under DirectRunner or in the cloud by switching to DataflowRunner and writing to BigQuery. The setup processes around 30 million rows (~1 GB) after a few years, the cloud run spins up dozens–hundreds of workers and completed in minutes for a one-off cost of about €3.4, so the approach works and is portable.
AI-written summary · about
chapters (8)
- 0:00Start
- 21:34What Is a Pipeline in Apache Beam
- 25:33Data Pipeline Options
- 29:24Maven Compiler
- 29:38Maven Compile
- 39:05From Batch to Streaming
- 41:02Visualize Local Pipelines
- 42:42Warnings
related talks
48mUsing Open Source Tech to Swap Out Components of Your Data Pipeline
16mYour Data Pipeline with Apache Beam
47mApache Beam - what do I gain?
21mPredicting Hacker News with Beam and TensorFlow
10mDemocratizing the pipeline with Apache Beam and Dataflow
19mApache Beam pipelines at 100TB+ scale using Apache Spark
46mBuilding an Apache Beam streaming runtime in GCP
41mApache Beam: Unified Data Processing