Petabytes of data: How to use R at scale
summary
Processes petabytes by moving work into a Spark cluster using HDFS/YARN, not by loading data into a laptop R session; a 200 terabyte CSV can be processed in-memory across nodes as Spark DataFrames. R starts a local driver that boots R on each node so sparklyr works but user-defined R functions are painfully slow unless bridged with Apache Arrow. Watch for lazy evaluation, driver OOM when pulling results, and reduceByKey vs groupByKey and data skew causing expensive shuffles.
AI-written summary · about
chapters (8)
- 0:00Introduction to talk
- 1:09Problem with scaling R
- 2:57Hadoop stack overview
- 4:03Apache Spark at scale
- 9:40Sparklyr and R integration
- 11:47Key Spark concepts
- 14:13Spark UI and debugging
- 15:36Resources and closing
related talks
11mScaling R with Spark
42mScalable Data Science in Python and R on Apache Spark
52mSolving Analytical Problems using Apache Spark
44mData Science at Scale with Apache Spark
30mSpeed Up Your Spark Data Processing with a GPU
50mIntroduction to Real-Time Big Data with Apache Spark
20mGive me the Data
03mFred Reiss Interview