From data science to scalable NLU and vision cloud service.
summary
Built a private NLU and vision cloud service handling 5M+ queries/month, implemented in Python with NumPy, PyTorch, TensorFlow, OpenCV, TorchVision, scikit‑learn and SQLAlchemy. Models live with their notebooks and model unit tests in the same repo to run in CI; a REST layer (OpenAPI/Swagger + Connexion/Flask) provides API key multi‑tenancy, dataset and hyperparameter ownership. Deployed containerised on Kubernetes with Rancher, Gunicorn and NGINX, backed by PostgreSQL/CloudSQL, meeting an average 10 requests/sec target and scaling by adding containers.
AI-written summary · about
chapters (18)
- 0:00Introduction and motivation
- 3:20Building our own NLU service
- 5:10Model testing pipeline
- 8:46Data scientist workflow
- 9:06Transfer learning overview
- 13:28Performance requirements
- 15:54Scalability challenges
- 17:02REST API architecture
- 21:13Switching to Flask
- 24:52Spec-driven development
- 26:19Monitoring with Prometheus
- 29:59Deployment on GCP
- 31:47Performance testing
- 32:08Q&A team size and mistakes
- 33:54Q&A model choices and languages
- 36:12Q&A POS tagging and data
- 37:15Q&A language learning software
- 38:23Q&A test automation
related talks
38mNatural Language Understanding at Scale
45mArchitecture of an NLP Deployment
35mCase Study: AI-as-a-Service on Kubernetes at Scale and In Production
40mBuilding highload ML powered service
17mDevOps for Private AI: A Scalable Platform Using Kubeflow
19mScalable Data and AI Solutions with Cloud-Native Architecture
30mAccelerating AI Application: NAVER's Journey from Model to Application
35mCloud-Native MLOps Platform for Python- and ML Engineers