Building scalable data pipelines through R and global health information systems' API
summary
Team builds reproducible, automated data pipelines in R to support USAID's $6 billion PEPFAR program, routinely pulling Datum and UNAIDS estimates via discovered APIs using glue, httr, and purrr. They keep it small by turning fragile clicks into parameterized functions, minimize the manual by replacing emailed Excel with Google Forms plus googledrive/googlesheets4, and package it up as ~15 R packages (Grabr, Glamr, Mind the Gap, HFR) used across 50+ countries, yielding faster, reproducible workflows.
AI-written summary · about
chapters (13)
- 0:00Rube Goldberg and data pipelines
- 1:48Manual workflows at USAID
- 3:57Automation vs accessibility
- 5:02Start small with APIs
- 7:55R package for API access
- 8:18Minimize manual steps
- 10:46Use existing infrastructure
- 13:15Package and document solutions
- 14:39Four lessons recap
- 15:43Simplifying data workflows
- 16:38Convincing leadership on APIs
- 17:41Training teams in data tools
- 18:44APIs vs R packages
related talks
35mProductionize your open-source project
25mA simple serverless data pipeline
37mSRE's Critical Role in the COVID-19 Pandemic Response in Government
14mRobust R Deployments: Building a Pipeline from RStudio to Production
43mA Year of Diagnosing Pediatric Cancer with Event-driven Architecture
38mUnleashing the serverless data pipelines
42mBuild and automate a modern serverless data lake on
17mCooperating Gault Tolerant Habitat #cheffriends