Exploring Tradeoffs in Experiment Management as part of AI Platforms
summary
Framework for deciding what to build versus buy for Experiment Management, applied to deployments across firms totaling $500B market cap and trading clients with $300B AUM. Presents a hosted service that uses an ensemble of Bayesian and global optimizers, parallelizes evaluations to cut tuning from 24 days → 3 days in one case, and supports distributed tuning, cluster management and hierarchical conditional parameter spaces to enable multimetric tradeoffs and unlock models previously too expensive to tune.
AI-written summary · about
chapters (8)
- 0:00Introduction and speaker background
- 1:04Purpose of SigOpt and mission
- 4:59Types of models and tooling needs
- 8:11Three key lessons for AI platforms
- 9:14Balancing flexibility and standardization
- 14:29Maximizing resource utilization
- 20:51Unlocking new modeling capabilities
- 24:01Closing remarks and Q&A
related talks
01h 03mExperimentation in Data Science
41mOptimizing Machine Learning Models
27mAutomating XGBoost Hyperparameter Tuning with Agentic AI
15mAI-Driven Application Optimization
29mContinuous tuning of Microservices via Bayesian Optimization
41mAutomated Performance Tuning with Bayesian Optimization
03mDBTB INT Scott Clark r
27mPlatforms for valuable AI Products- Iteration, iteration, iteration