Tips and tricks: data science prototype into production
summary
Solves a multi‑dimensional knapsack with tens of thousands of items, up to a hundred user constraints, using Google OR‑Tools and deployed to AWS Lambda; runtime wakes from SQS with inputs on S3 and a single entrypoint process allocation request that accepts pydantic models. Builds an 1,800‑line fixture suite, hits 87% coverage, and generates random synthetic loans for tests. Surprising specifics: informative errors returned to users, a mocked AWS cloud IO setup, and structured logging via a rotating file handler.
AI-written summary · about
chapters (12)
- 0:00Prototype to production tips
- 4:00Jupyter notebooks trap
- 5:03Single entry point for production
- 6:05Testing data consistency
- 7:08Pydantic for data validation
- 8:55Testing synthetic vs real data
- 10:38Code coverage and test patterns
- 12:02Handling failed requests
- 13:28Errors to informative responses
- 15:57Advanced Python logging
- 19:09Production-ready practices
- 20:14Q&A mocking and testing
related talks
17mProductionizing Data Science Workloads
40mDesigning Data Driven Projects: Beyond ML Experimentation
29mMachine learning in production
14mBattle-tested tactics to accelerate data science and ML R&D with AWS
31mProductizing Machine Learning over Big Data with AWS tools
36mApplied Machine Learning Playbook: From Idea to Production
30mOn scaling Machine Learning microservices
01h 02mUsing Python to Power Serverless Applications