Why Shipping Native Histograms to Our Users Is a Game Changer
summary
Claim: shipping native histograms inside Gloo gives much higher-resolution latency metrics at lower cardinality and uncaps the 100th percentile that classic Prometheus buckets hide. Implementation uses OpenTelemetry collectors feeding a dedicated Prometheus 3 instance with exponential/native histogram settings: bucket_factor, max_buckets, and periodic reset_duration. Example: a 60s top bucket collapses tail in classic histograms but native histograms reveal spikes and true p100. Migration strategy: side-by-side Prometheus scrape, dashboard queries using OR to fall back to old metrics; text exposition exporter support is still pending.
AI-written summary · about
chapters (12)
- 0:00Introduction and motivation
- 1:28Classic histograms limitations
- 4:42Native histograms advantages
- 6:27Instrumenting with native histograms
- 8:57Configuring bucket layouts
- 10:03Why native histograms are a game changer
- 12:11Use cases in infrastructure
- 14:40Migration strategy overview
- 17:34Phase one migration approach
- 20:44Dashboard comparison and insights
- 23:12Querying and visualization
- 23:32Q&A and closing
related talks
38mA Practical Guide to Prometheus for App Developers [B]
05mLightning Talk: Viewing Raw Samples for Troubleshooting Queries
06mTying a metrics to every release
32mHow Snowflake Migrated All Alerts and Dashboards to a...
05mLightning Talk: Improved Prometheus filtering in Grafana
34mPanel: OpenTelemetry: Bridging Platform and Enablement
42mEvolution of Observability Tools at Pinterest
26mProfiling Mastery: Decode Your Data and Elevate Observability