Building a Data Warehouse
summary
Describes building a cloud data warehouse using Google BigQuery, showing it can query a 1.5 TB (37 billion row) Wikipedia table in ~48 seconds and is billed by bytes scanned (only charged for used columns) with optional purchased BigQuery slots for fixed capacity. Covers ingestion choices (batch vs streaming), connectors such as Cloud Data Fusion, Airbyte, or custom API loaders, orchestration with Cloud Composer, federated cross-cloud queries, and design tradeoffs for update patterns (append/versioning vs frequent UPDATEs); recommends testing with free credits first.
AI-written summary · about
chapters (8)
- 0:00Data Warehouse intro
- 1:53Need a Data Warehouse?
- 8:15Building with Google Cloud
- 19:36How BigQuery works
- 24:55Migrating to BigQuery
- 32:00Data prep and loading
- 36:55Data analysis in warehouse
- 40:10Social initiative & closing
related talks
41mAnalyzing Terabytes of Data with Google BigQuery
33mBigQuery DeepDive: dissecting Google's Data Warehouse
52mBuilding the World's Largest Enterprise Data Warehouse with BigQuery
36mMigrating Our Data Warehouse to the Cloud
40mDeveloping on Google Cloud with Python and DataFrames
47mData Warehousing Migrations: Lessons from Home Depot
22mUsing BigQuery as a data warehouse in B2B startup
35mScaling Interactive and Insightful Dashboards with Data Studio and BigQuery