Simplified Data Quality Monitoring of Dynamic Longitudinal..
summary
A functional library of modular checks monitors dynamic longitudinal oncology data for over 2 million patients across ~20 cancer-specific datasets that are refreshed monthly with a 30 day recency. It composes small pure functions (using pandas nested DataFrames and partials) into higher‑order checks, wraps brittle steps with try/except, and speeds repeated work with functools lru_cache and a lazy MapReduce style over database chunks. The result is readable, testable, reproducible code that catches cross‑dataset regressions and tolerates missing columns.
AI-written summary · about
chapters (18)
- 0:00Data quality monitoring intro
- 1:02Speaker and company context
- 3:11Oncology data sources and scale
- 4:38Dynamic longitudinal data challenges
- 6:26Functional programming for data quality
- 9:35Nested data structures handling
- 12:05Nested data structures example
- 15:39Five benefits of functional approach
- 15:59Readable self-documenting code
- 17:01Compositionality in functions
- 19:10Reproducibility via immutability
- 20:16Robustness via exception handling
- 22:01Efficiency via lazy eval and caching
- 25:10Summary and resources
- 26:17Evolving data structures and exceptions
- 28:10QA reports cross-functional review
- 30:35Reactive vs proactive quality checks
- 32:00Adopting functional programming
related talks
01h 29mImproving Open Data Quality using Python
17mOpen Source for Data Quality
31mThe Hidden Costs of Data Quality - Tackling Common Data Challenges in ML
34mDataComPy - Dataframe Comparisons made Explicit
37mQuality Improvement with CQL and FHIR: Care Gaps and Near Real-Time Data Submission
01h 31mPandas, Data Wrangling & Data Science
35mBad data - anecdotes and examples from the real world
37mClinical Quality Improvement with FHIR