Spark Architecture
summary
Explains why Spark reduces repeated disk reads — typical Hive-on-MapReduce queries can cause 18 to 30 scans — and claims Spark delivers up to 100x speedup for some iterative ML workloads by caching intermediates. Describes RDD as metadata-only references, how shuffles force reexecution and originally created millions of files, then details hash, sort and Tungsten shuffles (Tungsten avoids deserialization using unsafe and serialized buffers) and outlines DataFrame as an RDD with schema and columnar storage for higher compression.
AI-written summary · about
chapters (11)
- 0:00Introduction to Spark
- 24:30Spark Architecture Components
- 26:55Nodes and JVM Details
- 30:30Spark Application Structure
- 32:05Data Persistence in Spark
- 38:11Spark Shuffle Mechanisms
- 40:10Sort Shuffle Implementation
- 41:44Tungsten Sort Optimization
- 44:12Data Frame Overview
- 47:01Data Frame Performance
- 51:09Q&A and Comparisons
