Frontier results, on device
summary
Comparative evals use Phoenix and production traces from Mima to show local models replacing frontier calls: testing Gemma, Qwen, Llama 3.2 against Claude Sonnet finds Llama 3.2 (3B) with a few‑shot prompt plus post‑processing hits 100% JSON validity, 100% structural validity and ~93% factual consistency, P50 ≈1s and P95 under 3.5s, while saving roughly $1/day in inference. The process: prototype on a foundation model, build a golden dataset, test small→large, iterate prompts and harness code.
AI-written summary · about
chapters (15)
- 0:00The hidden costs of AI
- 1:10Analyzing inference expenses
- 2:51Introduction to SLMs
- 4:33Evaluating local model performance
- 6:52Local AI in practice
- 8:47Framework for right-sizing AI
- 10:47Setting up golden datasets
- 12:58Model selection process
- 15:55Finding the 'Sage' model
- 19:22Optimizing with prompt engineering
- 21:12Iterative prompt testing
- 23:04Closing the performance gap
- 25:40Post-processing for reliability
- 27:45Regression and maintenance
- 28:36Next steps for local adoption
related talks
31mFrontier results, on device
Stop Paying for Frontier Models
12mDemo: Taking Gemma from prototype to production faster with Vertex AI
12mDemo: Rapid prototyping with Gemma and Llama.cpp
01h 45mRun Frontier AI at Home
44mBuilding amazing Front-End Project with Gemma: A State-of-the-Art Open Model
21mFrom 46% to 90%: Fine-Tuning Tiny LLMs for On-Device Agents
02h 52mLow Level Technicals of LLMs