Ka-Ching! How Enterprises Measure LLM Performance and Cost
summary
A working POS demo logs ~50 real prompts (33 KB) and feeds them into an SLO-driven benchmark pipeline that uses Guide LLM against a local VLM inference server running a Granite 8B model; a 30s throughput run yields ~6 requests/sec, median inner-token latency ≈42 ms, P95 about 10% slower, and time-per-request for 256 output tokens ~56 s while time-to-first-token stays under target 200 ms. Model accuracy checks use LM Eval Harness (ARC score ~81%), and quantization reduces model size to make these results achievable on commodity GPUs.
AI-written summary · about
chapters (9)
- 0:00LLM benchmarking intro
- 7:052025 LLM evaluations
- 12:48LLM performance metrics
- 20:05GuideLLM SLO benchmarking
- 22:35GuideLLM live demo
- 34:43LM Eval Harness accuracy
- 40:25Model optimization with quantization
- 43:13Unit testing LLM outputs
- 47:31Q&A
related talks
30mEfficiently Deploying and Benchmarking LLMs in Kubernetes
16mHow fast are LLM inference engines anyway?
10mRunning LLMs locally: Practical LLM Performance on DGX Spark
29mLeveraging open-source LLMs for production
24mLLM Inference Optimization From Simple To Sophisticated
23mObservability for LLMs: Understanding What’s Happening Under the Hood
51mPractical LLM Inference in Modern Java
31mMy AI is slow. Make it faster!