Effortlessly modernize your legacy
summary
Merck migrates large PySpark pipelines to DuckDB using SQLFrame converts PySpark to DuckDB SQL, keeping S3 and job orchestration unchanged while switching execution to single-node instances with NVMe SSD spill. Median scan sizes ~55 MB and 99.9th percentile ~35 GB justify no distributed engine. Results show roughly 75% less compute across SAP transformations and one job with an 11,000-line SQL reduced runtime from 7h to under 1h, enabling much faster, simpler production runs.
AI-written summary · about
chapters (6)
- 0:00DuckDB in enterprise
- 2:06Legacy challenges and PySpark limits
- 3:32Migrating to DuckDB with SQLFrame
- 5:38SQLFrame transpiles PySpark to DuckDB SQL
- 8:27Performance after DuckDB migration
- 10:15Key takeaways and next steps
related talks
40mDuckDB beyond the notebook
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
40mDuckDB: Harnessing in-process analytics for data science and beyond
27mIn Process Analytical Data Management with DuckDB
25mProcessing Biggish Data with DuckDB and Python
37mDuckDB: Crunching Data Anywhere, From Laptops to Servers