Making Data Pipelines in R: A Story From A “Self-Taught” Perspective

on YouTube ↗20m

" summary

Built a working R pipeline for a local health department that ingests messy opioid datasets (including a Narcan field coerced as “4 plus”), automates cleaning, validation, and HTML report rendering, and documents flow on GitHub (https://github.com/Meghansaha/pipelines_in_R). Uses modular scripts and a docking script to orchestrate sourcing and conditional processing, adopts project-aware file structures with the here pattern, adds assertions via pointblank, and publishes a readable data workflow map to make maintenance and handoff feasible.

AI-written summary · about

" chapters (7)

1. 0:00Data pipelines in R
2. 1:51First encounters
3. 6:30Environmental structures
4. 10:52Internal environments
5. 14:10Data validations
6. 16:43Pipeline sustainability
7. 19:13Recap and key takeaways

" related talks

• 20mMaking Data Pipelines in R: A Story From A “Self-Taught” PerspectiveMeghan Harris · rstudio::conf(2022)
• 16mAutomating the Dutch National Flu Surveillance for Pandemic PreparednessPatrick van den Berg · posit::conf(2023)
• 34mData pipelines powered by Open source and funSayantika Banik · stackconf 2022
• 43mData Pipelines à la ModeTom Hall · Code Mesh LDN 2019
• 30mDesigning Resilient Data PipelinesAndrew Bolin · SREcon19 Americas
• 26mSimplifying Data Analytics Using RudderStack Open Source Data PipelinesSumanth Puram · OSA Con 2021
• 41mAccelerating privacy-enhancing data processingFlorian Stefan · PyCon Italia 2025
• 14mDigging a Pit of Success for Your OrganizationAaron Chafetz · rstudio::conf(2022)