Scaling Generative AI Inference at Trillion-Token Scale
summary
Claims handling hundreds of billions of tokens per day, a proprietary FP8 quantization that preserves quality while speeding execution, and a homegrown continuous fetching algorithm for late-arriving and early-finished requests. Platform components include Friendli Suite with container images and managed dedicated endpoints, one-click deploy from Hugging Face, autoscaling and replica routing. Notes agentic AI drives up to 10x API call growth, explains batching vs. caching trade-offs, and emphasizes GPU-cost reduction through kernel optimizations and cache-aware batching.
AI-written summary · about
chapters (9)
- 0:00Introduction and speaker background
- 1:59Rising inference demand drivers
- 3:25Challenges in AI inference scaling
- 4:50Friendly AI platform overview
- 6:36Optimization principles for LLMs
- 8:19Batching and caching trade-offs
- 10:49Friendly Suite product demo
- 12:16Live deployment walkthrough
- 12:38Promotion and closing remarks
related talks
20mOperating Frontier Intelligence at Scale
26mAmazingly Fast and Incredibly Scalable Inference...
20mFireworks AI: Go from Hype to High - Value AI
24mScalable GenAI in Production
25mThe Golden Triangle of Inference Optimization: Balancing Latency, Throughput, and Quality
13mScaling AI Systems for High-Volume Transaction Processing
27mGen-AI at Scale: Simplifying Orchestration of Healthcare Applications Across Multi-Cluster Environme
32mScaling AI: Fault-Tolerant GenAI Solutions for Millions