Python & DAG architecture: Winning combination for dev of complex algo agile flows

on YouTube ↗26m

" summary

Shows how Python (including libraries like Tickle and Dask) is used together with Directed Acyclic Graph (DAG) architecture to manage algorithm development in the semiconductor domain. Describes treating each algorithm as an independent Python package/node that receives inputs/parameters, and composing those nodes into flexible DAG flows so units can be developed, updated or reused with minimal impact on others in a continuous delivery ecosystem.

AI-written summary · about

" related talks

• 29mDagster: a Framework for Data Processing ApplicationsNick Shrock · Scale By The Bay 2019
• 47mDagster: An open source Python library for building data applicationsNicholas Schrock · Crunch Conference 2019
• 27mModern ETL-ing with Python and Airflow (and Spark)Tamara Mendt · PyCon.DE 2017
• 46mThe Python Data Ecosystem: Navigating a fragmented landscapeUmare & Tong · PyData Seattle 2023
• 34mPython as a Framework for Analytics and Growth HackingIgnacio Elola · PyData Berlin 2015
• 34mData Ingestion in Python using Apache AirflowSowmya Ganapathi Krishnan, Yew Wee · PyCon APAC 2018
• 38mAirflow DAGs Made Easy: Introducing the Astro SDKDaniel Imberman, Tamara Janina Fingerlin · Python Web Conf 2023
• 01h 25mApache Airflow in the Cloud: Programmatically orchestrating workloads w/ PySatyasheel, Kaxil Naik · PyData London 2018