Battle-tested tactics to accelerate data science and ML R&D with AWS
summary
Claims a lightweight R&D pipeline that wraps a simple single-data-point Python preprocessor with an orchestrator over S3 and compute so teams move from days to minutes for large video/image preprocessing. Uses S3 + Lambda orchestration and elastic EC2 workers instead of heavy Airflow setups, supports retries, logging and online data validation, and lets teams reuse tested building blocks so code moves to production with minimal friction. Reports speedups using 1,000 servers in parallel.
AI-written summary · about
chapters (10)
- 0:00Intro
- 0:44Data scientists lack tooling to be more effective - Immature tooling ecosystem
- 2:25A typical day for a computer vision engineer • Running experiments requires processing massive amounts of data
- 3:13Production-grade data pipelines aren't the solution - Practitioners often run multiple new experiments daily
- 5:08Optimizing the code isn't enough - Workloads might be too large for a single server, even when optimized
- 6:29Our guiding principles Minimal overhead learning curve
- 8:43Package proprocessing as a function that manipulates a single dieta pont
- 9:10Submit job along with input and output data locations
- 9:27behind the scenes)
- 10:52Successful outcomes - Increased productivity
related talks
48mAccelerate science, research, and innovation with AWS
34mBuilding scalable end-to-end deep learning pipelines in the cloud
32mServerless Data Processing and Machine Learning
31mProductizing Machine Learning over Big Data with AWS tools
25mDesigning an AWS Well Architected Framework for AI/ML
17mProductionizing Data Science Workloads
12mLeveraging AWS and Kubernetes for Scalable AI/ML Workloads in the Cloud
46mServerless for Data Scientists