Do those numbers look right? How to ensure quality, and avoid inaccuracies in your data insights.
summary
Public Health England underreports about 2,000 cases a day for eight days is a concrete failure mode that motivates building tests into data pipelines. The session shows refactoring Fabric notebooks into functions and running unit tests with in‑memory Spark DataFrames so an omitted duplicate-drop fails fast; using Synapse pipelines with async polling, Gherkin specs and snapshot testing (serialize output JSON) to catch regressions; and executing DAX via the Power BI Execute Queries REST API or local Analysis Services port to validate tabular models before deployment.
AI-written summary · about
chapters (5)
- 0:00Introduction
- 13:28Unit testing in Fabric notebooks
- 20:47Integration testing with Synapse pipelines
- 31:23User acceptance testing in Power BI
- 43:32Key takeaways and closing thoughts
related talks
25mTesting Data Pipelines
23mChallenges of database testing
18mData quality: the key to long term happiness
28mHow to not pull your hair out while providing data to the business
50mNo more data quality surprises in your Databricks, Synapse or Power BI solutions!
20mShift-left testing : Building reliable Data Pipelines
39mWhat Can Go Wrong With Testing an ETL Pipeline?
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines