How We Built a Metering & Chargeback System to Incentivize Higher Resource Utilization
summary
Built a platform-agnostic chargeback metering pipeline that ingests Aurora/Mesos, Hadoop and Manhattan usage via ETL, normalizes 500+ metrics into a time-varying unit-price model, and emits monthly bills and Tableau reports. Key pieces are a service identity model, a resource catalog, a Data‑Fertility checker and a resolved fact table. Rollout takes 2–4 weeks per infra. Result: quota drops while utilized cores rise about 33% year over year, 100M+ meter rows stored, and concrete savings (one team cut $200k/month).
AI-written summary · about
chapters (42)
- 0:00Intro
- 0:27Agenda
- 1:14World Cup 2010
- 3:11Microservices
- 6:38Platform Engineering
- 9:41Platform Services
- 10:14Understanding the Use of Resources
- 11:05Chargeback System
- 13:30Service Identity
- 13:52The Problem
- 15:14The Solution
- 16:02The End Result
- 16:16Service Identity Identifier
- 17:05Service Identity Example
- 17:53Resource Catalog
- 19:58Resource Fluidity Example
- 20:36GPU Network Capture
- 20:48Accountability
- 21:39Unit Prices
- 23:51Entity Model
- 24:34Metering Pipeline
- 25:13How the System Works
- 27:40What is Infrastructure
- 28:51Budgeting Report
- 29:28Custom Reports
- 30:36Case Study
- 33:55Unified Compute Cloud Platform
- 35:59Thank You
- 36:32Do we have tooling
- 36:45Do we use tableau
- 37:28Aurora
- 38:19Service Identification
- 38:54Team Management
- 39:34How did this inspire engineers
- 41:42Onboarding a service
- 42:45Recap
- 44:06Chargeback Amount
- 44:27Income
- 47:05Public Cloud
- 47:27Kite vs Public Cloud
- 48:40Metrics
- 49:48Data Storage
related talks
29mHow We Built a Framework at Twitter to Solve Service Ownership & Improve Infrastructure Utilization
27mHow to get rich with cloud!
46mWhen the CFO Comes Knocking: Our Journey to a Cost Efficient Data Platform
10mLightning Talk: Show Me the Money: Metrics Edition
33mHow we do cloud cost efficiency at Notion
42mA Year with Apache Aurora
40mCloud Costs Panel "Lots of Reward Points"
36mFinOps & Observability: technical implementation