The Birdwatcher’s Guide to Optimised Tabular Data Pipelines

on YouTube ↗25m

" summary

A real pipeline walk-through: a naive O(T·D) loop on 5 million detections and 300 species becomes a linear-ish sweep-line with sorting, dropping an XS case from ~20s to <1s and the large pipeline from ~9 minutes to 36s after successive fixes. Key swaps are using Polars (lazy execution, Arrow memory) and columnar Parquet inputs, switching loops to vectorized expressions, using categorical enums, and streaming; GPU offload via Rapids/QDF fails to help. Memory falls from ~14GB to ~9GB.

AI-written summary · about

" chapters (33)

1. 0:00Introduction
2. 0:21Talk overview
3. 1:02Aircraft pipelines to bird counts
4. 2:50Pipeline setup
5. 3:51Naive algorithm baseline
6. 4:36One sweep line optimization
7. 5:39Small dataset benchmarks
8. 5:59Vectorization with pandas
9. 6:41Large dataset benchmarks
10. 7:02Choosing data frame library
11. 8:49Why Polars is optimal
12. 9:11Narwhals compatibility
13. 10:13Polars speed with Rust and Arrow
14. 11:16Polars expressions and lazy execution
15. 12:43Lazy execution analogy
16. 13:24Lazy execution in code
17. 14:06Projection and predicate pushdowns
18. 14:49Lazy execution with Iceberg
19. 15:32Choosing data types
20. 16:58Categorical and enum types impact
21. 17:18Selecting file formats
22. 18:45File formats for analytics
23. 19:51Parquet encoding benefits
24. 20:12File format benchmarks
25. 20:35Streaming for memory efficiency
26. 20:56Streaming query limits
27. 21:18Streaming performance gains
28. 21:39GPU acceleration
29. 22:02Polars GPU performance
30. 22:45Final pipeline results
31. 23:30Memory usage improvements
32. 23:50Optimization checklist
33. 24:12Closing and bonus footage

" related talks

• 37mTaming the Data TsunamiDavid Aronchick · PyData Seattle 2025
• 35mData Processing Pipelines for Small Big DataEsteban Gabancho · PyData Global 2020
• 24mTaming Data Skew in Production ML PipelinesSrihari Godleti · Conf42 Machine Learning 2026
• 32m7 steps to build-your-own data pipeline - for day 1 of your startupKumar Puspesh · The Fifth Elephant 2019
• 18mData Preprocessing for ML at Production ScaleWenlei Xie, Vitaly Fedyunin, Yingxin Kang · Data @Scale Spring 2022
• 03h 07mData of Unusual Size: Interactive VisualizationPavithra Eswaramoorthy, Dharhas Pothina · PyCon US 2023
• 33mData preparation and analytics on your laptop with DuckDBMatson, Prawiroatmodjo · PyData NYC 2024
• 40mDuckDB: Harnessing in-process analytics for data science and beyondGabor Szarnyas · Øredev 2023