The Software Engineering Part of Data Science
summary
Recommendations are concrete: use Docker for notebooks and one credential per data scientist, enforce pre-commit hooks (flake8, pylint), and manage dev vs prod deps with poetry. Build CI/CD that auto-generates docs and packages on merge. Benchmark models — one tested model hits 1000 inputs and saturates CPU, another blows memory — so avoid keeping models permanently in memory, make them pickle-safe, prefer lighter inference frameworks, target O(n log n) time/space, and choose edge vs cloud by resource profile.
AI-written summary · about
chapters (19)
- 0:00Intro
- 0:38Welcome
- 1:33Agenda
- 2:54Data Science vs Software Engineering
- 6:40Software Engineering Challenges
- 8:46Working with Data Scientists
- 10:18Collaboration Setup
- 12:03Environment Management
- 13:44Code Quality
- 15:43Class User
- 16:18Handling Errors
- 17:40CICD
- 21:48Production Setup
- 22:07Target Environment
- 22:51Time Space Complexity
- 23:18Platform Hardware dependencies
- 23:44Data scientists are not software engineers
- 24:06Advice for software engineers
- 24:40Conclusion
