How to structure PySpark application
summary
Shows a concrete PySpark app layout: extract input/output and app name to a config file, add a main entrypoint that loads a job from a jobs package, move UDFs to a shared module, and register third‑party libs via a requirements.txt and zipped site‑packages deployed with --py-files. Uses a Makefile (make build) to produce a dist/ with job and shared zips, runs tests with pytest and pytest-spark plus pandas.testing, deploys artifacts to S3 via Terraform and CI, and runs reliably on AWS Glue/EMR for ~200 small jobs.
AI-written summary · about
chapters (16)
- 0:00PySpark introduction
- 1:42PySpark structure matters
- 3:11Spark submit setup
- 4:15First ETL jobs
- 5:41Extracting configs
- 7:29Main entry point
- 9:16Refactoring structure
- 10:24Makefile packaging
- 12:16UDFs for data ops
- 14:03Third-party dependencies
- 16:54Testing importance
- 18:46Automated tests
- 19:10Final structure deployment
- 21:23Structured approach benefits
- 24:44Q&A: AWS Spark jobs
- 26:44Q&A: Debugging workflow
related talks
43mPySpark in Practice
44mUsing Python with Hadoop to create production ready big data applications
52mBlurring the line between Developer and Data scientist with PixieDust
46mSparking Success: Unveiling the Journey of Apache Spark Application Development
37mPixieDust make Jupyter Notebooks with Apache Spark Faster, Flexible, and Easier to use
01h 12mBuild & deploy a Snowflake Native Application using Python
24mPySpark - Data processing in Python on top of Apache Spark.
01h 22mIntroduction to Big Data Processing using Spark and Python