The Right Model for the Right Job: Cutting LLM Costs Without Cutting Quality
summary
Demo moderates high volume soccer chat with an 8‑label classification set and runs a 16‑model benchmark to pick the cheapest model that meets a quality bar. Evaluation uses MLflow + Unity Catalog to log traces and LLM judges, with automatic and human review. Production examples: a customer switches to a fine‑tuned open‑weight model and achieves an 86% drop in cost per task. Small models can match accuracy — Gemma 12B = 0.85 accuracy at $23 vs GPT4o nano 0.85 at $912 — and fine‑tuning lifts accuracy 0.85→0.91 for narrow classification. Smart routing and AI gateway handle model selection and budgets in production.
AI-written summary · about
chapters (11)
- 0:00Why Do AI Agents Become Too Expensive to Scale?
- 3:40How to Reduce LLM Costs Without Sacrificing Quality
- 5:45Can Open-Weight Models Cut AI Costs Dramatically?
- 9:54How Do You Choose the Right LLM for Each Task?
- 11:28How to Build an LLM Evaluation Baseline With MLflow
- 14:35How to Compare Multiple LLMs Using the Same Evaluation
- 17:42Why Model-Agnostic Architecture Matters for AI Applications
- 19:15How to Use Smaller Models Without Losing Accuracy
- 22:18When Is Fine-Tuning an LLM Actually Worth It?
- 25:55How Smart Model Routing Balances Cost, Quality, and Performance
- 30:01How to Reduce a High LLM Bill: Models, Caching, and Memory
related talks
30mLLMs: Beyond the Hype - A Practical Journey to Scale
25mSelecting the Best LLMs with Model Cockpit
40mScaling generative AI workloads with efficient model choice
53mLLM Quality Optimization Bootcamp
39mThe Future is Fine-Tuned: Deploying Task-specific LLMs
20mLLM: Efficiency and Sustainability in Practice
21mFrom Data to Precision: Fine-Tuning LLMs for Proprietary Query Languages
28mNavigating LLM deployment: Tips, tricks, and techniques