Cloud warehouse monitoring ( Snowflake case study )
summary
Snowflake case study shows a monitoring system built from account logs and Airflow metrics that identifies runaway compute: a single Excel-style warehouse loads ~150 GB/day, one table is queued 128,000 times in five months, tasks fail 400 times/day, and a 1,200-line SQL runs repeatedly — invoices jump from ~1k to ~100k. The build publishes org-wide dashboards, exposes task creators, enforces role lockdown and CI/CD, and reduces costs to 20%, saving millions.
AI-written summary · about
chapters (16)
- 0:00Snowflake adoption journey
- 3:20Snowflake misuse
- 6:31First invoice shock and monitoring
- 9:01Optimization reports and best practices
- 12:22Excel Warehouse hidden costs
- 14:33Spending explosion and politics
- 16:20Service accounts and legacy SQL
- 18:34Data science cost visibility
- 21:08SQL certification and failures
- 23:16Task history and role cleanup
- 25:23Admin restrictions and cost tracking
- 27:09Refactoring and governance
- 28:13Cost reduced to 20%
- 28:56CI/CD in production
- 30:02Forced certification pushback
- 31:32Q&A
related talks
28mBuilding A Governed Data Lake
13mLightning Talk: Highlights of the Snowflake Elastic Data Warehouse
43mData warehousing reinvented for today's needs
28mScalable AI & Web Insights with GitLab, Snowplow & Snowpipe
32mScalable Python and SQL Data Engineering without Migraines
27mHow To Develop & Operate An Agile Cloud Datawarehouse In A Rapidly Changing Multi-ERP Environment
32mHow open source is re-shaping the cloud data warehouse landscape
41mRole of Private Data Lake for Sustainable Business Growth