Applying Data Quality Monitoring
summary
Claims that data observability reduces time to discovery and resolution are backed by concrete steps: map metadata and assign ownership, run profiling to collect baselines, convert recurring patterns into reusable validation templates, schedule checks or gate ETL pipelines, and use lineage plus logs/traces/metrics to root‑cause. Tools mentioned include Selectzero, Monte Carlo, dbt, Databricks Gen AI, DataHub and Soda; real incidents cited (Kinnect energy €50M error, Google 20% PLN rate glitch) show failures are costly and that integrated validation and LLM‑assisted SQL generation are already deployed.
AI-written summary · about
chapters (11)
- 0:00Data quality introduction
- 1:10Why data quality matters
- 3:59Data quality dimensions
- 8:16Standards and quality checks
- 12:13Poor data consequences
- 14:00Data observability channels
- 17:57Static vs dynamic observations
- 20:30Investing in quality frameworks
- 21:59Implementing quality framework
- 26:55Emerging tools and automation
- 29:51Conclusion
related talks
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
37mEngineering data quality
05mData Quality and Validation in ML Pipelines
41mData Observability - System Health to Business Health
40mBuilding trust in your data. Why data governance is the key to success.
41mBeyond the Dashboard: Building Trustworthy Data with Observability at PAYBACK
41mData Quality: The Holy Grail For A Data Fluent Organization
47mData governance