The COVID Scenario Pipeline: High Stakes Data Science
summary
A volunteer team spins up a reproducible pipeline that turns the Johns Hopkins SEIR code into docker images, profiles and speeds it to run in minutes, and then scales to 10,000 simulations across ~150 counties. They implement an MCMC-enabled forecasting loop that checkpoints to S3 and runs on hundreds of machines using spot instances, discover a DVC checkpointing bug that breaks convergence, fix visibility, and deliver timely county-level planning scenarios.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 0:51Welcome
- 1:45Disclaimer
- 3:14Personal feelings
- 6:54Epidemiology
- 10:23Impact of Non Pharmaceutical Interventions
- 12:16The First 90 Days
- 14:21Open Source Tools
- 15:42Lift and Shift
- 16:58Special Note of Thanks
- 17:57Running Models in the Cloud
- 19:53DVC Data Version Control
- 21:04Davis Batch
- 22:50Spot Instances
- 23:56Visibility
- 25:13Tools
- 25:27sharpen your axe
- 26:16Thank you
related talks
31mAn incomplete list of implementing data science effectively
22mCOVID-19: Data Science & Expertise
29mHow data science shortened the COVID-19 pandemic by 2 months
21mUsing R to develop production modeling workflows at Mayo Clinic
44mML in Production
45mWorkshop: Deploy and Monitor ML Pipelines with Open Source and Free Applications
34mData Products, From 0 to data science pipeline
24mUncovering Community and Project Insights Through Data Driven Methods