My AI is slow. Make it faster!
summary
Benchmarks show big differences: time to first token and total time matter more than topic, with some models returning in seconds while others spike up to a minute on the same prompt (example: Llama 3.2 responds in seconds, another takes ~54s). Use the LLM CLI and the llm-profile plugin to measure chunks per second and draw box‑and‑whisker variability. Practical fixes: shorter responses, distillation and quantization, semantic caching (embedding similarity) and model routing to route cheap queries to faster models.
AI-written summary · about
chapters (7)
- 0:00AI performance challenges
- 7:45AI response time expectations
- 13:20Measuring AI performance
- 19:23LLM CLI performance testing
- 23:16Optimizing AI responses
- 27:05Speeding up AI models
- 30:17Summary
related talks
50mKa-Ching! How Enterprises Measure LLM Performance and Cost
19mReal-Time AI: Low Latency Solutions for Interactive Applications
10mRunning LLMs locally: Practical LLM Performance on DGX Spark
03mFinetune Llama 4 Faster
24mCost and Performance Optimization of LLM Inferencing.
48mPracticalities of Fine-Tuning Llama 2 with AI Studio
15mAI Trends: Insights
01h 45mRun Frontier AI at Home