10 things I learned about writing data pipelines in Python and Spark.
summary
Built a production Python + Spark pipeline that collates ~200 data points and computes ~300 metrics for ~80M account filings from ~11M private companies, uses mapping dictionaries to call small metric functions, applies decorators and inspect to handle zeros and nulls, and relies on dict unpacking and mapping dictionaries and partials to keep computation tidy. It runs on a 128‑node cluster, runs weekly taking 2–3 days, and is expanding to 12 European countries after practical handling of many real‑world data gotchas.
AI-written summary · about
chapters (15)
- 0:00Data pipelines in Python & Spark
- 1:08What to expect from this talk
- 2:36Who should attend
- 3:17Doodle’s financials pipeline
- 9:19Clean code for metrics
- 13:17Data cleaning pitfalls
- 16:12Division errors & type checks
- 18:02Dictionary unpacking in functions
- 22:00Mapping dictionaries for metrics
- 23:44Trend metrics & sliding windows
- 25:37Python closures gotchas
- 27:06Floating-point precision issues
- 29:18Importance of specifications
- 30:01New structural projects
- 31:28Q&A on data sources
related talks
50mBig Data Processing and Data Science Environment in Python
39mHow Python Drives Modern Data Engineering
24mPySpark - Data processing in Python on top of Apache Spark.
01h 22mIntroduction to Big Data Processing using Spark and Python
46mSparking Success: Unveiling the Journey of Apache Spark Application Development
27mModern ETL-ing with Python and Airflow (and Spark)
05mBig Data, No Big Deal: PySpark
40mScaling Python - Bank Edition