Lightning Talk: Ditching Data Pipelines: why treating Data as Assets is the best...
summary
Argues that replacing step-by-step pipelines with asset-based lineage simplifies modern cloud data platforms, claiming Dagster infers dependencies just by returning and referencing data objects rather than coding orchestration. Shows concrete integrations: DBT for transformations, TensorFlow for models, Airflow/Jenkins as legacy contrasts, and Kubernetes plus S3/object storage for runtime. Highlights Dagster’s IO managers that swap local Docker storage and cloud S3 without changing pipelines and an ecosystem of integrations that make lineage visible out of the box.
AI-written summary · about
related talks
41mModern Data Architectures – What They Are and How to Implement?
29mDagster: a Framework for Data Processing Applications
14mA Brave New World: How New Lineage Technology Can Reduce Data Spend 40%
43mData Pipelines à la Mode
01h 28mA Guide to the Modern Data Stack
30mPlatform Oriented Data Architecture for Acceler...
34mTales of Data Architecture Evolution
35mCloud Native Data Platforms - The why, what and how