Mapping FHIR data to parquet for analytics and machine learning
summary
Maps FHIR JSON to Parquet by defining an explicit Parquet schema, batching records into configurable row groups to trade memory for compression, and converting choice types, recursion and extensions by either expanding fields or serializing deeper/variable parts as JSON strings. An open source pipeline writes Parquet files to Azure Storage partitioned by resource and date, creates Synapse SQL pool external tables and views to flatten arrays, and uses Spark to train a LightGBM model on the Parquet data with good query performance on selected columns.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 5:11Mapping data to parquet
- 8:27Gaps to handle
- 12:03Load data critically
- 13:35Analyze data
- 22:08Machine Learning
- 24:32Summary
related talks
35mScalable Data Science with FHIR
44mTransforming FHIR Data for horizontally scalable analytics
39mHow to integrate FHIR with your analytics pipeline
58mHow to transform AVRO (IDL) data to multiple PARQUET files
33mBridging Big Data and Machine Learning Ecosystems: A Cloud Native Approac...
46mEfficient Data formats for Analytics with Parquet and Arrow
49mAzure Synapse Analytics
33mA 101 in time series analytics with Apache Arrow, Pandas and Parquet