Notorious B.I.G. Data: Wins and fails when building a big data platform in the cloud
summary
Startup scales from hundreds of rows/day to 3.5 billion rows/day and grows to 25+ paying clients while moving from SQL scripts on BigQuery to a Dataflow clustering pipeline, then to a hybrid that does a group-by-key in BigQuery and runs clustering in Dataflow to cut costs by ~95% on grouping. Initial approaches work fast but become expensive or slow; the final design reduces cost, enables internal productization, and increases modularity at the cost of complexity.
AI-written summary · about
chapters (5)
- 0:00Start
- 5:25Prerequisite Information
- 14:45Present day - Third Visit Architecture
- 18:03Empower teams!
- 18:45Philosophies
related talks
36mMigrating Our Data Warehouse to the Cloud
33mManaging IoT Storage with Google's Cloud Platform
52mBuilding the World's Largest Enterprise Data Warehouse with BigQuery
25mData Demons: Unleashing Chaos or Driving Performance?
41mAnalyzing Terabytes of Data with Google BigQuery
22mUsing BigQuery as a data warehouse in B2B startup
45mBuilding a Data Warehouse
36mBig Data, Disruption And The 800 Pound Gorilla In The Corner