Save an ocean of time: streamline data wrangling with R
summary
Develops an R package called strings R package to automate compiling data from 250+ ocean sensors that produce ~4 million rows per year, cutting processing time by 95%. Uses tidyverse tools and RMarkdown for QC, visualization and reports; adds checks that flag positive longitudes and mismatched sensor serials so wrong files are caught before compilation. Compiles 71 deployments in 11 weeks (a task that previously required ~9 months) and replaces a two‑year manual workflow.
AI-written summary · about
chapters (8)
- 0:00Writing code for data wrangling
- 3:18Automating sensor string data
- 6:56Time saved by automation
- 7:58Reducing data errors with code
- 11:18Traceability and reproducibility
- 12:43Code versus manual work example
- 13:45Starting small with automation
- 14:26Final thoughts
related talks
15mSave an ocean of time: streamline data wrangling with R
08mProcessing Robot Data at Scale with R and Kubernetes
20mMaking Data Pipelines in R: A Story From A “Self-Taught” Perspective
17mBuilding a data science pipeline for the FT with RStudio Connect
26mSimplifying Data Analytics Using RudderStack Open Source Data Pipelines
16mEXCELsioR!
22mData Science in Meatspace
24mManaging Heterogeneous Spatial Data with Analysis Ready Cloud Optimized formats