How to spend ¾ of your yearly budget in 3 weeks: a PySpark cautionary tale
summary
A recommendation engine built with PySpark on Databricks reads orders then 1000x larger member pageviews from Snowflake and S3 and runs cosine similarity across a huge sparse matrix and accidentally spends three quarters of a team budget in three weeks. The job exhibits intermittent Snowflake JDBC connector rereads that hit an unexposed 60-minute S3 temp timeout, cache calls materialize unexpectedly, and default partitions blow up; fixes: monitor credits, test small, repartition (~256MB), write intermediates to S3, and push vendors on hidden timeouts.
AI-written summary · about
chapters (9)
- 0:00PySpark cautionary tale intro
- 1:25Problem and product approach
- 4:58Switching to pageviews data
- 7:47Unlimited budget spending
- 9:57Six PySpark lessons learned
- 16:48JDBC connector pitfalls
- 18:33S3 token expiration cause
- 23:48Key suggestions recap
- 24:56Q&A
related talks
38mPySpark and Warcraft Data
39m10 things I learned about writing data pipelines in Python and Spark.
43mPySpark in Practice
18mETL at Scale: Optimizing PySpark + Airflow Workflows
43mImproving PySpark Performance: Spark performance beyond the JVM
38mHow to incrementally scale existing workflows on Spark, Dask or Ray?
25mDistributed data workflows: PySpark vs Dask
05mBig Data, No Big Deal: PySpark