Building Trust in Your Data - Improving Your Data Quality with Automated Framework on AWS
summary
Automated data quality framework called Project Soteria enforces 13 standard checks (including z‑score drift tests and population sampling), runs checks on a separate Redshift snapshot cluster to avoid production impact, and stores results in Postgres while exposing a JupyterHub workspace and an in‑house DBT‑like pipeline layer for analysts. Ownership of rules moves from engineers to dataset owners, cutting disruptive on‑call firefights and enabling plans for self‑healing fixes plus more advanced statistical checks to preserve long‑term trend comparability.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:24Trust in data matters
- 2:49Data quality failure story
- 5:56Manual checks challenges
- 8:43Overwhelming alerts impact
- 11:51Democratizing automated frameworks
- 13:59Project Soteria design
- 16:47Soteria security and scalability
- 17:54Framework for new analysts
- 19:40Self-healing techniques
- 20:02Conclusion and Q&A
related talks
28mHands-on Workshop: How To Perform Data Quality At Scale
24mMetadata driven framework for data discovery and data quality
30mData Engineering building your BI infrastructure from scratch
18mData quality: the key to long term happiness
29mManaging your Data Assets: The interview
21mLarge-scale Data Quality Verification (...)
37mData Validation at Scale: Managing Data Quality in Complex Data Pipelines
15mTrust your data with data integrity on (PRT055)