Building the World's Largest Enterprise Data Warehouse with BigQuery
summary
Claims BigQuery removes the table-scan constraint via Dremel-style distributed execution over Colossus and Jupiter and adds dynamic query planning to choose hash or broadcast joins at runtime. Demonstrated features: column-based partitioning and clustering that cut scans dramatically; TPCDS/TPC-H runs show about 5x faster and a petabyte retail query falling from roughly $5,000 to ~0.001 USD. Also announced SQL MERGE, customer crypto-delete pattern for PII, and self-service quota controls.
AI-written summary · about
chapters (23)
- 0:00Intro
- 2:05Databases; how do they work?
- 2:49Databases; how do they scale?
- 3:51The Dremel Paper
- 4:03Dremel: An alternative approach to scaling
- 4:32Database Design Constraints
- 4:59Removing Constraints
- 5:34Dremel Architecture (2010)
- 6:06Dremel Architecture (2018)
- 10:38Ingestion time-based partitioning
- 11:42Column-based partitioning
- 12:12Clustering
- 18:15Security
- 20:15BigQuery and VPC
- 22:43Using Encrypted Columns
- 23:20Crypto Delete
- 28:462D Information vs 3D Information
- 35:14Traditional DWH Model
- 38:35Aggregate UDFs Follow the Pattern
- 39:12Example Data
- 39:46Demo
- 47:06A simple example
- 48:13Nested Data is Co-located Data
related talks
06mGoogle BigQuery for enterprise
41mAnalyzing Terabytes of Data with Google BigQuery
45mBuilding a Data Warehouse
33mBigQuery DeepDive: dissecting Google's Data Warehouse
36mMigrating Our Data Warehouse to the Cloud
25mData Demons: Unleashing Chaos or Driving Performance?
22mUsing BigQuery as a data warehouse in B2B startup
34mGoogle BigQuery