Benchmarking GenAI Like a Pro: Scaling Experiments, Predicting Performance, and Keeping Your Sanity
summary
A reusable benchmarking stack is built and open-sourced as ADO (github.com/ibm/ADO) that runs tens of thousands of GenAI experiments across models, hardware and configs using Ray for scale, Pydantic schemas with MySQL persistence and a kubectl-like CLI. Average fine‑tuning benchmark time drops from ~1 hour to ~6 minutes via early-stopping heuristics, and predictive models need only ~20–40 new samples to predict a new software or GPU variant within system noise.
AI-written summary · about
chapters (7)
- 0:00Benchmarking GenAI intro
- 3:56GenAI benchmarking challenges
- 10:55Unified benchmarking framework
- 15:26Accelerated Discovery Orchestrator
- 21:02FMS actuator fine-tuning
- 24:35Predictive scaling models
- 28:51Call to action & OSS release
related talks
44mKeynote: Tracking the Pulse of Generative AI
53mHow to Build and Scale Ultra-Lean Gen AI Startups
28mOptimizing GenAI with Intel Gaudi Accelerators on
44mKeynote: Tracking the Pulse of Generative AI
40mScaling generative AI workloads with efficient model choice
16mBenchmarks Are Memes: How What We Measure Shapes AI—and Us
23mScaling GenAI Applications and Agentic AI for Business Competitive Differentiation
36mNavigate the future of AI: Deploy generative models on Amazon EKS