Tamland: How GitLab.Com Uses Long-Term Monitoring Data For Capacity Forecasting.
Talkat KubeCon + CloudNativeCon North America 2022 · on Friday, 2022-10-28· at 16:00· Virtual Platform Only·
summary
A capacity-forecasting pipeline called Tamland pulls normalized Prometheus recording rules (utilization ratio 0–1) from Thanos long-term metrics, aggregates by max per service/resource, and runs Facebook Prophet in Python inside Jupyter Book via GitLab CI to produce weekly static reports. It forecasts three months ahead across about 360 service-resource combos, the report job runs ~90 minutes, and the system identified a needed short-term fix that allowed a multi-month Postgres decomposition to proceed without saturation. Forecasts usually work but miss very recent change points until reviewed.
AI-written summary · about
chapters (18)
- 0:00Intro
- 0:23A series of Postgres saturation events..
- 4:14Definitions: Resources
- 6:35Saturation Events vs. Mitigation Time
- 8:10System Capacity Planning
- 10:04Load Testing Maximum Capacity
- 11:23Guessing Maximum Capacity
- 12:45Modelling Maximum Capacity
- 13:16Estimating System Capacity is Hard
- 14:10Resource Capacity, not System Capacity
- 15:24Resource/Service Matrix
- 16:35Measuring Utilization Consistently
- 17:11Prometheus Recording Rules
- 19:16Aggregating to Reduce Data
- 21:10Real-time Utilization Monitoring (Detail)
- 22:18Using Thanos for Long-Term Metrics Storage
- 22:57First Attempt at Forecasting: Linear Regression
- 28:57Runaway Session State Bug
related talks
01h 12mDetecting Workload Anomalies with Prometheus and Machine Learning
20mHow We are Dealing with Metrics at Scale on GitLab.com
37mPractical Capacity Planning using Prometheus
37mAIOps: Anomaly Detection with Prometheus and Istio
27mObservability with Prometheus and beyond
34mEfficient Resource Management
36mObservability at Scale With Monitoring as Code...
01h 02mThe Power of Metrics, Logs & Traces with Open Source