Dashboards & Dragons: Reliability Magic for AI Platforms
summary
Bloomberg describes a multi-cluster GenAI platform instrumented with SLOs that target token streaming: an example SLO is time to first token (TTFT) ≤ 500 ms at 99.9% over a rolling one‑day window (0.1% error budget). Dashboards show token throughput, GPU saturation, inter‑token latency and grouped 5xx/4xx errors, backed by distributed tracing for agentic call chains and LM‑as‑judge checks. Alerts use error budget burn rate and time‑to‑exhaust; dashboards and traces prove useful to pinpoint bottlenecks.
AI-written summary · about
chapters (15)
- 0:00AI platform reliability intro
- 2:44Crafting SLOs for GenAI
- 6:15Measuring GenAI latency
- 9:00Four ruins of reliability
- 11:50Designing AI dashboards
- 13:14Dashboard walkthrough
- 15:21Token throughput & GPU saturation
- 16:46Debugging request bottlenecks
- 18:55Resource management for GenAI
- 19:59Agentic systems & tracing
- 22:45Meaningful alerts from SLOs
- 25:54Tuning SLOs via incidents
- 27:41Enriching dashboards with context
- 29:25Data-driven reliability decisions
- 30:07Closing & career tips
related talks
31mDashboards & Dragons: Crafting SLOs To Tame the AI Platform Cha...
01h 00mPrompt Failures and Latency Spikes: Observability for AI
26mAI-Driven Cloud Observability: Reduce toil and amplify time to resolution
16mOperationalizing AI in Finance: Real-Time Analytics
29mBuilding Resilient AI Platforms: Financial Fraud Detection
38mAI Monitoring Mastery: Optimize Performance, Ensure Compliance, and Cut Costs
25mAn SLO-Driven Approach to Enhance Kubernetes Cluster Reliability
30mReliable Observability at Scale: Error Budgets for 1,000+