From Pandas to PySpark
summary
Hands-on roadmap for taking pandas code to PySpark using an anime recommendation dataset in Colab, showing concrete commands to install PySpark, create a SparkSession, load CSVs, infer and explicitly set schemas, and convert back to pandas. Demonstrates reading cleaning and transforming millions of records with timing notes (cells that take minutes for actions), builds an ETL-like pipeline (joins, explode, casts), explains lazy transforms vs actions, and shows optimizations: caching, broadcast joins and repartition to fix skew — all run end-to-end in the notebook.
AI-written summary · about
chapters (21)
- 0:00Workshop introduction
- 1:50Transition to PySpark
- 4:19Spark and MapReduce background
- 8:21Spark vs Hadoop and Excel
- 14:55PySpark setup in Colab
- 22:18Spark session and data load
- 28:11Schema inference in PySpark
- 31:50Data type changes and cleaning
- 33:13Transformations vs actions
- 34:22Spark architecture
- 39:50Lazy evaluation and DAGs
- 41:57Data cleaning and nulls
- 49:31Immutability in PySpark
- 55:35String cleaning and manipulation
- 57:46Shuffling and performance
- 1:00:20Broadcast joins and optimizer
- 1:02:50Spark SQL API
- 1:04:56Catalyst optimizer and caching
- 1:05:59Switch back to pandas
- 1:07:24Use cases and scaling
- 1:10:12Frequently asked questions and closing
related talks
01h 22mIntroduction to Big Data Processing using Spark and Python
25mScale Data Science by Pandas API on Spark
45mIndroduction to the PySpark DataFrame API
50mBig Data Processing and Data Science Environment in Python
02h 52mFrom Spreadsheets to DataFrames
24mPySpark - Data processing in Python on top of Apache Spark.
44mIntroduction to Pandas
01h 27mPorting Existing Python and Pandas Code to Spark, Dask, and Ray