Working with Data Scientists and why I left my job (not)
summary
Three years of cross‑functional teams at Prima move data scientists from notebook chaos into production-safe workflows using git branches and required one approval and CI pipelines that block merges until checks pass. Notebooks never go to production; models and artifacts become services, developed as packages on Databricks. Enforce style with black autoate and rough and pre-commit hooks, test by mock small realistic datasets and validate models on ~5–10 points plus end-to-end pricing checks to catch regressions.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 1:29Cross-functional teams at Prima
- 2:55Version control for data scientists
- 6:01Testing in data science
- 8:08Code quality checks
- 10:38Educating data scientists
- 11:45Moving away from notebooks
- 13:30Gradual adoption of practices
- 14:58Making it fun with memes
- 16:46Don't leave your job
- 17:13Q&A session
related talks
41mBetter Code for Data Science
40mThe data scientists are gone. how do I deploy their work to prod?
05mProductionalizing Data Science Models
41mCan Data & ML really learn from software engineering?
17mProductionizing Data Science Workloads
59mDevOps for Data Science
37mPatterns for Collaboration Between Data Scientists and Software Engineers
29mWhen Worlds Collide: Productionalizing a Data Science Model