When Less Is More: A Technical Overview of LLMs and the Strength of Smaller Models
summary
Claims concrete: quantizing FP16 to INT4 via LLM Compressor halves inference time (example: 4s → 2s) while keeping 99% accuracy, and distillation plus sparsity also reduce footprint. Production stack uses vLLM with SafeTensors from Hugging Face, GuideLLM for system benchmarks and LM Eval Harness for model comparisons. Assertion: an 8B on-prem setup is cheap enough to run locally for privacy and agentic chains of specialized SLMs.
AI-written summary · about
chapters (14)
- 0:00The Rise of SLMs: Technical Overview of Small Models
- 0:38Pre-training Data: How Common Crawl Powers Modern AI
- 1:08Data Hygiene: Filtering PII, Duplicates, and "Bad Results"
- 2:07Tokenization 101: Converting Human Language to Binary Symbols
- 3:05Inside the Neural Network: Weights, Parameters, and Loss Reduction
- 3:47Vector Embedding Space: Visualizing High-Dimensional Logic
- 4:33Production Inference: Deploying Models with vLLM Engine
- 5:21The Deployment Paradox: Balancing Latency, Accuracy, and Cost
- 6:23Defining "Small Models": Optimal Parameters for Local Dev
- 6:42Quantization Deep Dive: Shrinking Models Without Losing Accuracy
- 7:45Performance Benchmarks: FP16 vs. Compressed Inference Speeds
- 8:40Local Sovereignty: Data Privacy and On-Prem Stack Control
- 9:27Designing Agentic Workflows: Chaining Task-Specific SLMs
- 10:12Engineering Toolkit: Red Hat AI Repos, GuideLLM, and Benchmarking
related talks
30mLLMs: Beyond the Hype - A Practical Journey to Scale
21mScaling LLM Inference with vLLM
38mOptimizing vLLM Performance through Quantization
40mSurviving the Agentic Hype with Small Language Models
24mLLM Inference Optimization From Simple To Sophisticated
30mAccelerated LLM Inference with Anyscale
19mQuant-LLM: Accelerating the Serving of Large Language Models via FP6-Centric...
29mLeveraging open-source LLMs for production