Writing your own RDD for fun and profit
summary
Explains how to implement custom RDDs by coding the three required pieces: getPartitions, compute and parent dependency links, plus optional data locality and partitioner hooks; shows a simple RandomRDD and a CensorRDD built on top of HadoopRDD and run via SparkContext.runJob; demonstrates that narrow vs wide dependencies control shuffles, that compute provides the per-partition iterator, and that mutating data in compute can break later actions — examples compile and run from staged commits.
AI-written summary · about
chapters (12)
- 0:00Start
- 0:53Recap
- 6:32Internals
- 7:21How Data Is Internally Partitioned
- 14:58Wider Dependency
- 15:30Narrow Dependency
- 16:01How To Evaluate Internal Data
- 19:11Method Count
- 23:18Data Locality
- 25:31Partitioner
- 32:18Compute Function
- 37:35Implicit Conversion
related talks
50mIntroduction to Real-Time Big Data with Apache Spark
55mSpark Architecture
52mSolving Analytical Problems using Apache Spark
47mKeeping the “fun” in Apache Spark Datasets and FP
35mDataFrames in Apache Spark - the analysts perspective
29mBig Data Analysis
01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live coding
43mOptimizing Spark applications from simple to complex with examples