Pipes or Brackets: Dplyr and Data.Table
summary
Recreates payroll timekeeping from six messy sources into a unified table of 102 million rows and shows concrete timings: pure data.table loads and aggregates that data about six times faster than dplyr and fread reads large files in minutes versus read.csv in hours. Combined workflow keeps data.table for joins and in-memory ops using := in-place assignment and uses dplyr/tidyr for complex text/date parsing, producing a pipeline that runs far faster than dplyr alone while accepting the tradeoff of data1[data2] joins and denser, harder-to-read syntax.
AI-written summary · about
chapters (9)
- 0:00The R data debate
- 2:10Understanding Dplyr
- 4:00Advantages of Tidyverse
- 8:06Introduction to Data.table
- 11:00Data.table performance
- 14:18Syntax comparison
- 17:57Real world case study
- 22:54Combining Dplyr and DT
- 25:07Q&A and conclusion
related talks
20mBridging the Gap between SQL and R
18mInteractivity and Programming in the Tidyverse
16mWorking with names and expressions in your tidy eval code
30mData Rainbows - select * from cloud;
21mDon’t repeat yourself, talk to yourself! Reporting with R
16mEXCELsioR!
21mHow to use Pandas Efficiently
38mKeeping pace with data infrastructure so it doesn't outrun you