Argonauts of Data: Building Scalable and Effective Data Pipelin...
summary
Atlan reworks ETL by splitting the heavy column transformation work into chunked 10M files and using Argo’s withSequence plus a semaphore to parallelize transforms, cutting a 100M‑row transform from ~22 hours to a fraction and claiming a 450% speedup. Subsequent runs use marker‑based incremental extraction to fetch only changed metadata, dropping extract times to minutes. Dynamic resource sizing plus incremental retries reduce failures and raise success rate from ~78% to 96.5%, and swapping PySpark for DuckDB processes millions of query histories far faster.
AI-written summary · about
chapters (10)
- 0:00Atlan and Argo workflows
- 2:32Why Argo workflows for data
- 3:58ETL workflows in Argo
- 4:39Performance challenges at scale
- 5:00Optimizing first-run ETL
- 9:07Fixing subsequent-run issues
- 12:19Reliability for resilience
- 13:22Dynamic pod allocation
- 16:49DuckDB parallel queries
- 23:13Summary and Q&A
related talks
06mA quack at building scalable data pipelines with DuckDB
10mScaling Time Series Analysis With Argo Workflows: Patterns and Practi...
29mLarge Scale Batch Processing with Argo Work...
31mOne Data Pipeline to Rule Them All
18mOur Analytical Data Journey – from Paper to Petabytes
29mBuilding a Batch Processing Platform...
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
25mRemove Slow, Expensive ETL & Bring Data To The Data Lake For Lower Cost, More Performant Analytics