Big Data, No Big Deal: PySpark

on YouTube ↗05m

" summary

Demonstrates using PySpark to turn a single Washington Post police shootings CSV into a distributed dataset by writing it partitioned by state in parquet so Spark can run queries across many files, then creating a PySpark DataFrame that executes distributed selects, filters, group by and SQL and returns aggregated results from each partition; notes demo files remain local but scale on cloud, and shows Koalas adds pandas methods like value_counts to bridge gaps with pandas.

AI-written summary · about

" chapters (6)

1. 0:00Intro
2. 0:41Apache Spark
3. 0:57Distributed Data
4. 1:37Data Source
5. 2:26Data Frame
6. 3:39Koalas

" related talks

• 50mBig Data Processing and Data Science Environment in PythonRick Bahague · Pycon PH 2017
• 24mPySpark - Data processing in Python on top of Apache Spark.Peter Hoffmann · EuroPython 2015
• 01h 22mIntroduction to Big Data Processing using Spark and PythonRaoul-Gabriel Urma · PyData NYC 2018
• 01h 12mBig Data in Java World: Apache Spark and Scala. Arch and live codingOleg Klimenko · Java ON 2022
• 01h 36mSpark for Scala DevelopersBrian M Clapper · Scala World 2017
• 41mKeynote: Making the Big Data ecosystem work together with PythonHolden Karau · PyData London 2018
• 45mIndroduction to the PySpark DataFrame APIPeter Hoffmann · PyData Berlin 2015
• 52mSolving Analytical Problems using Apache SparkRohit Bhardwaj · Developer Summit 2020