FHIR R4 data ingestion pipelines driven by liquid templates
summary
Northwell runs a production FHIR R4 repository ingesting messages since 2020, storing ~2–3 TB and covering ~6.5M patients with 0.5–2M FHIR bundles ingested per day and 100k–300k patient data updates daily. The pipeline uses liquid templates (Python library) for JSON mapping, Dataflow (Apache Beam) for streaming vs scheduled transforms, and BigQuery validation to check accuracy. Custom code-lookup filters, conditional inserts, cleaning rules and Cloud Scheduler restarts handle duplicates and malformed HL7 sources; the system is live and monitored.
AI-written summary · about
chapters (7)
- 0:00Introduction to FHIR pipelines
- 2:54Challenges in FHIR data ingestion
- 7:08Liquid templates for transformation
- 11:44Handling HL7 message variations
- 14:12Lessons learned and validation
- 16:44Monitoring and alerting system
- 18:32Conclusion and Q&A
related talks
19mFrom Zero to Production FHIR Server in 6 Months at a Large Northeast Healthsystem
16mThe story of FHIR data exchange and storage combined
45mGoogle Cloud FHIR APIs: Data Ingestion, Management, and Analytics
21mClinical and Claims Data Transformation utilizing FHIR Bulk Data
44mTransforming FHIR Data for horizontally scalable analytics
20mPrototyping data ingestion pipelines
48mAnalytics and AI on FHIR
38mScaling FHIR with microservices