Cloud Analytics: from 0 to 100,000 events per second in 50'
summary
Builds a complete cloud data analytics pipeline that handles 100,000 events per second, from event ingestion to real-time reporting. Uses Google Cloud technologies — Pub/Sub for ingestion, Dataflow for processing, BigQuery for storage/analysis, Data Studio for reporting, and demos on Datalab and Compute Engine — with tips, lessons learned and live coding.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 2:14Cloud data examples
- 8:39Cloud generations
- 10:25Big data history
- 14:57Google data stack
- 18:47Google ecosystem products
- 21:15Customer cases
- 28:15Lambda architecture
- 31:41Pub/Sub BigQuery demo
- 35:56BigQuery political demo
- 41:56Real-time analytics
- 44:28100k events demo
- 52:22Summary
related talks
24mGive meaning to 100 billion analytics events a day, analytics at Teads
20mStreaming Processing Options in Google Cloud
19mLog Events @ Twitter: Challenges of Handling Billions of Events per Minute
41mReal-time Data Processing with Cloud Dataflow
39mLarge scale streaming analytics
26mETL pipeline with GCP Dataflow
30mData Warehouse in Google Cloud
20mBuilding gaming analytics online services with Google Cloud and Improbable