Recipe for Building a Discoverable and Governed Data Platform.
summary
Claims a working recipe: uses DataHub to catalog Snowflake, Looker, Terraform, Airflow, Kinesis and custom configs, and builds column-level lineage that enables impact analysis when a column changes. Implements a push-based pipeline with crowd-sourced metadata extractors and a central metadata API, plus a protocol buffer SDK that scrapes schema comments and annotations into tags and classifications. Adds a Looker Chrome plugin, assigns owners, enforces governance via glossary-to-policy mappings and runtime assertions to form data contracts; the system scales beyond a single platform team.
AI-written summary · about
chapters (16)
- 0:00Introduction
- 1:14Data tool explosion challenge
- 2:37Conway's law in data teams
- 3:41Bring teams to one platform
- 5:26DataHub metadata platform
- 6:28Common data language with schemas
- 8:37Schema as documentation tool
- 8:59Logical and physical data layers
- 11:32Separate governance from data
- 14:21Crowdsource metadata injection
- 17:07Assign data ownership early
- 17:50Meet users in their tools
- 18:55Shift left cataloging to producers
- 20:39Data contracts for producers
- 23:11Recap of 10 tips
- 23:53Q&A on LLMs and governance
related talks
39mCrafting modern data platforms: reveal your data value
35mBuilding a data analytics platform on strong opinions held loosely
27mDataOps & Full Snowflake Automation In 1 Click
28mBuilding A Governed Data Lake
25mPerfecting Our Modern Data Stack Recipe: What We Learned On The Journey At Huel
17mDemocratising data analytics
28mUsing Dataops To Build And Manage Data Products On Snowflake
25mBuild open format data lakehouse