Federated Data, Centralized Action: A Governance Model Powered by Lakewatch and Databricks System Tables
summary
Claim: a three‑pillar pipeline turns Databricks System Tables into continuous, actionable governance by extracting usage, aggregating with Lakewatch and scoring in Cortex to drive cost and best‑practice remediation. Built artifacts: dbt transformations, medallion layers, Unity Catalog governance, an MCP server and GitHub Copilot hookup. Pilots run three months, one team’s score moves from 58% to 92%, sizing recommendations use 95th‑percentile CPU/memory heuristics, and only aggregated metadata leaves the Databricks ecosystem. Databricks System Tables Lakewatch Cortex scorecards
AI-written summary · about
chapters (15)
- 0:00Federated data architecture intro
- 1:50ASML’s federated architecture
- 3:14Centralized data challenges
- 5:00Federated governance pillars
- 7:06Federated architecture problems
- 8:09DataBricks system tables
- 10:16LakeWatch transformation
- 12:45Cortex integration scorecards
- 15:31Benefits and use cases
- 19:27Self-remediation empowerment
- 21:14Pilot results progress
- 23:42Chief Data Office role
- 26:10Security and governance
- 26:52Future plans AI
- 27:55Q&A
related talks
51mAir Traffic Control (Data Governance) with Databricks Unity Catalog, tips, tricks and best practices
42mDriving Data Democratization
28mBuilding A Governed Data Lake
28mLakehouse Architecture with Delta Lake and Databricks
49mDelta Live Tables: Building Reliable ETL Pipelines with Azure Databricks
41mBeyond the Dashboard: Building Trustworthy Data with Observability at PAYBACK
43mUse dbt and DuckDB Instead of Databricks for Data Processing
19mRobust CI Workflow with dbt for Data Warehouse Optimization