Looking under the hood - demystifying data tools
summary
Benchmarks compare R Dataframe, R data.table, Pandas, Numpy, PySpark RDDs, PySpark Dataframes and RedShift on reads, group-by, merges and joins and show concrete takeaways: R data.table fastest overall for in-memory SQL-like ops, numpy is slow reading raw CSVs but blazing when using .npy serialized files, Pandas trades some speed for heterogenous convenience, R data.frame occasionally beats expectations on a log-scaled plot, Spark DataFrames outperform RDDs in-memory and RedShift leads on large aggregations and joins; memory limits and CPU utilization drive the move to distributed tools and the repo contains the benchmark code.
AI-written summary · about
chapters (18)
- 0:00Start
- 0:34Data comes in different shapes and sizes So pick your data tool wisely
- 1:51Diving in ...
- 2:47Pandas' Blocks Manager
- 3:53NDArrays under the hood of Pandas
- 4:52Slicing Pandas DF - View
- 5:29Under the hood of R
- 6:19Data Operations
- 8:24Inner Join/Merge
- 8:57Sort with log on x-axis
- 10:30Moving to Big Data realm is inevitable
- 11:23Aggregation
- 12:00Merge/Join
- 13:22Key Takeaways - Numpy
- 13:53Key Takeaways - Pandas
- 14:36Key Takeaways - Redshift
- 15:13Key Takeaways - Spark
- 15:49Cost comparison
related talks
30mSpeed is Not All You Need for Data Processing
19mSpark vs. Accelerated Compute: A Balanced Look at Data Processing Strategies
01h 08mUsing and contributing to the data.table package for efficient big data analysis
40mBeyond Pandas: The Great Python Dataframe Showdown
43mIntroduction to Spark Datasets
39mMastering DataFrame Diffing Techniques
24mAn Introduction to PySpark
44mProcessing large data with Apache Spark