Save an ocean of time: streamline data wrangling with R
summary
Organization runs over 250 oceanographic sensors producing about 4 million rows per year. A custom R package called strings package standardizes imports, enforces QC (coordinate sign checks, expected sensor serials), and drives RMarkdown reports. Manual Excel processing would take roughly two years; automation cuts processing time by about 95%, lets an intern compile 71 deployments in 11 weeks, and detects misfiled sensors and bad coordinates before analysis.
AI-written summary · about
chapters (8)
- 0:00Writing code for data wrangling
- 3:18Automating sensor string data
- 6:56Time saved by automation
- 7:58Reducing data errors with code
- 11:18Traceability and reproducibility
- 12:43Code versus manual work example
- 13:45Starting small with automation
- 14:26Final thoughts
related talks
15mSave an ocean of time: streamline data wrangling with R
08mProcessing Robot Data at Scale with R and Kubernetes
20mMaking Data Pipelines in R: A Story From A “Self-Taught” Perspective
16mDo It For Yourself: Creating a data input platform using R
14mDigging a Pit of Success for Your Organization
16mEXCELsioR!
24mManaging Heterogeneous Spatial Data with Analysis Ready Cloud Optimized formats
16mAutomating the Dutch National Flu Surveillance for Pandemic Preparedness