You got your engineering in my Data Science: Addressing the reproducibility crisis
summary
Argues that adopting software engineering practices actually reduces the reproducibility crisis and cites concrete failures: fMRI pipelines flag spurious results ~70% of the time, psychology replication ~36%, landmark cancer papers ~11% reproducible. Recommends specific tools and practices: git and Git LFS for data tracking, build scripts and repositories for repeatable transforms, CI/CD plus unit tests and fuzzers to catch regressions, and Zenodo DOIs for archiving code; internal anonymization services and inner-sourcing improve corporate reproducibility.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 4:38Reproducibility crisis
- 10:17Engineering lessons for data science
- 17:15Automated testing and validation
- 23:15Open source and data sharing
- 27:47Modern tools and gradual improvements
- 29:10Q&A
related talks
31mYour Research Code Works.. Until It Doesn’t: Rethinking Reproducibility
30mA data scientist's guide to code reviews
41mBetter Code for Data Science
39mRight Code, Right Place, Right Time
01h 14mUp your Bus Number - A Reproducible Data Science Workflow
44mBehavioral Software Engineering
05mThe Five Principles of Data Science Education
22mBridging Business and Tech for Data Science Success