Can Data & ML really learn from software engineering?
summary
Argues that software practices help but don’t map one-to-one: notebooks are for experiments not production, test logic with unit and integration tests but add data tests at runtime because content changes break pipelines, and CI/CD for ML means continuous training plus deploying model artifacts. Recommends Git for versioning, DBT for declarative data tests and observability, and keeping subsets or anonymized copies of production data in non‑prod environments so pipelines and models validate on real signals. Continuous contracts make teams talk.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:51Misconceptions about data engineering
- 3:37Data in modern products
- 4:42Software basics for data scientists
- 10:05Differences between data and software
- 13:40When data becomes software engineering
- 21:51Software practices for data teams
- 27:12Testing data pipelines
- 30:00Data contracts and collaboration
- 32:55Continuous integration and deployment for data
- 37:10Balancing software in data teams
- 39:38Why engineers should embrace data
related talks
35mCan Data & ML really learn from software engineering?
28mDataEngBytes 2023 - PER-T1-05 - Ellie Gholami
26mCompleting the machine learning loop
33mAI with a DevOps mindset - experimentation, sharing ..
06mMachine learning is about your data and deployment
21mHow The Application Of Machine Learning Is Changing The Way Companies Use Data
17mContinuous deployment for ML: The new software development lifecycle
31m🤑 Paying Off Technical Debt in the Development of Machine Learning Apps