Accelerating privacy-enhancing data processing
summary
Shows a concrete pipeline that turns messy hospital records (free text, scanned documents, multiple standards) into research-grade datasets by chaining an extract/normalize/load library into a warehouse, SQL transforms and post-DB NLP, all orchestrated without heavy infra. Uses Dexter, dbt and DT with Ibis and Snowflake (or DuckDB for local/hospital runs), cuts warehouse costs by ~50%, supports NLP/OCR harmonization, implements consent/deletion bookkeeping, and surfaces results used in ~1,200 publications and ~9,000 citations.
AI-written summary · about
chapters (11)
- 0:00Introduction and agenda
- 1:52What is real world evidence
- 5:26Challenges in data integration
- 7:33Data architecture overview
- 10:22Feedback loops in data systems
- 13:05Technology stack overview
- 20:53Building the data pipeline
- 29:43Patterns to accelerate feedback loops
- 31:48Making pipelines portable
- 35:44Impact on cancer research
- 37:11Q&A session
related talks
05mDump the Ducktape: Unblocking data pipelines at scale in healthcare using
40mProtecting sensitive data in huge datasets #bbuzz
37mA novel way to manage and represent unstructured clinical data
34mData pipelines powered by Open source and fun
01h 17mData Engineering in the AI Era: Building a Robust Foundation for AI Applications
44mTransforming FHIR Data for horizontally scalable analytics
07mDBTB INT Andy Petrella
47mWhat it takes to process healthcare data?