Build, Buy and the Golden Ratio: How Theator scaled up its continuous training framework
summary
Company scales from 80 videos to 15,000+ procedures, ingesting hundreds more daily while automating nightly retrains and closed‑loop feedback. Uses ClearML for dataset versioning, experiment tracking and orchestration; models run in PyTorch on NVIDIA GPUs (training on NVIDIA 300s, edge on Javier HDXs, cloud T4s) with DeepStream 5 for inference. Built a pipeline master controller DSL to spawn Kubernetes jobs, auto-scale GPUs, manage complex model dependencies, and push zero‑code research results to production.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:59Build vs Buy
- 5:08Hidden Technical Depth
- 12:20Data
- 13:30Research
- 16:12Continuous Training
- 18:20Lessons Learned
- 20:42Summary
- 22:20QA
related talks
34mDeep Learning at Scale
27mHarnessing Kubernetes for Scalable AI/ML Workloads
13mFrom Silicon to Scale: Training at Scale
12mKeynote: AI Infrastructure
22mBuild and Monitor Machine Learning Services in Kubernetes
42mDeep learning beyond the learning
21mOperationalizing ML Training Infra at Meta Scale
21mHow Ray and Anyscale Make it Easy to do Massive-scale ML on Aerial Imagery