Taming the Cost of Telemetry: How Riot Games Reined In...
summary
Riot ingests ~3.9 petabytes/month into telemetry and finds ~40–60% unused; migration to DataDog agents + Vector collectors plus team rules (metric graduation PRs, aggressive log sampling, EVPF defaults) cuts waste. A single change — blocking zero-valued metrics at the vector layer — yields ~99.9% cardinality reduction for one service; company-wide log ingest drops ~80%, monthly overspend alerts catch thousands in leakage, and cost attribution ties bills to service owners.
AI-written summary · about
chapters (11)
- 0:00Introduction to Riot Games
- 2:45The telemetry cost problem
- 5:13Assertions and principles
- 7:34Migration and organization
- 9:51Standards for waste control
- 13:15Real-world optimization stories
- 17:51Accountability and attribution
- 21:22Alerting and prevention
- 22:38Lessons and outcomes
- 25:37Operational threats
- 28:35Final notes and conclusions
related talks
26mRiot Games: Evolution of Observability at the Gaming Company
06mObservability Diet: Your 5-Step Plan To Trim the Data Fat
30mPanel: Telemetry That Matters
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
27mThe Signal in the Storm: Practical Strategies for Managing Telemetry Overload
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
30mBut Wait! There's...Still More‽ - Observability Data Volumes and Strategies for Managing Them
32mThe Invisible Tax: How Data Format Conversions Drive up Telemetry Pipeline Costs