Inference in Progress… Please Monitor Responsibly
summary
Running GPU inference at scale uses a control plane plus compute plane with a cluster agent (NVCF/NVCA) that reports GPU types and orchestrates worker pods; monitoring centers on inference SLOs (success rate) and time to first token with drilldowns from global endpoint → region → cluster → node. Concrete tools and signals: DCGM exporter, GPU Operator, NFD/GFD/device-plugin labels, NV Sentinel for auto-remediation, and traces/metrics; NVCF claims 500+ gov sites, 15M daily and 6B total inferences proving the platform at scale.
AI-written summary · about
chapters (14)
- 0:00AI inference observability
- 3:12Inference platform architecture
- 7:46Reliability with SLOs
- 10:32GPU node health
- 16:12Inference request performance
- 18:18Open-source tools
- 19:44Q&A
- 21:14Invocation plane and servers
- 23:46Beyond time to first token
- 26:55Multi-model routing
- 29:03Distributed tracing
- 30:45Model vs hardware bottlenecks
- 31:28Multi-cloud healthcare models
- 35:39NVIDIA Cloud Functions
related talks
41mPanel: AI Infra Best Practices: Enterprise Do’s and Don’ts
20mOperating Frontier Intelligence at Scale
24mCluster Management for Large Scale AI and GPUs: Challenges and Opportunities
35mUtilization Is the Key to Efficiency: What It Takes to Run...
06mAI Inference at the Edge: Beyond the Hype
07mYour GPU delivery is here! Now what? A 5 minute guide to make your CTO happy
25mOpen Source Meets Production MaaS: GMI Inference Engine and Karmada for Global AI…
04mA Cloud Native Stack From Bare Metal To Tokens for Large-Scale AI Inference