Running serverless jobs on GCP

on YouTube ↗51m

" summary

Shows how to build a serverless streaming pipeline on Google Cloud Platform by wiring Pub/Sub topics into a Dataflow (Apache Beam) job that computes flight arrival and departure delays and writes results to BigQuery. Demonstrates autoscaling limits (Dataflow capped at 150 workers in the demo), cost examples (~€90/month for continuous Dataflow), Cloud Functions free tier (2M invocations), and exposing results via a Data Studio dashboard and BigQuery live-view.

AI-written summary · about

" chapters (9)

1. 0:00Start
2. 2:36Serverless Architecture
3. 4:58Storage
4. 12:19Streaming data ingestion: Pub/Sub
5. 13:26Ingestion and pipeline: Dataflow
6. 16:59Pipeline: Cloud Functions
7. 23:19Data exploitation
8. 25:22Final architecture
9. 27:54Pro mode: automate batch tasks

" related talks

• 38mServerless tools for machine learning, big data and app backendsBret McGowen · DevFest Ukraine 2018
• 54mFocus on Your Code, Not Infrastructure, with Google Cloud FunctionsMartin Omander · Windcity DevFest 2019
• 31mServerless datastream processingMartin Gorner · Devoxx Belgium 2017
• 07mServerless: Accelerate your AISara Ford, Debi Cabrera · Google Cloud Next 2024
• 37mIntelligent Serverless and Scalable Real-Time Data Pipeline using Kinesis, Fargate and CFNYogesh Sharma · DataEngBytes 2021
• 50mGo Serverless with Google CloudMarek Biskup · GDG DevFest Sofia 2019
• 49mGet silly with serverlessBret McGowen, Guillaume Laforge · Devoxx Belgium 2017
• 47mServerless Compute at the Heart of Your EDAJulian Wood · GOTO Copenhagen 2024