~/talk

The Great Cardinality Disasters of Our Time

on YouTube 39m
Talkat KubeCon + CloudNativeCon North America 2019 · on Wednesday, 2019-11-20· at 17:20· Room 17AB - San Diego Convention Center, Mezzanine Level· [observability, devops-and-delivery]

speakers

name:Bryan Boreham
github:bboreham
name:Chris Marchbanks
links:none on file

resources

video:https://youtu.be/6f9z9JSNOGk
slides:none on file

summary

Prometheus deployments hit real limits: queries over 100,000 series slow or time out, a single server struggles past ~10 million series, and mislabels can generate 300,000 alerts or write 80,000 samples every 15s. Long error strings explode the TSDB symbol table (one case from 5GB to 60GB), and mtail misconfiguration follows 50,000 log files until relabeling reduces that to 79. Use TSDB analyze/promtool, relabeling, admin delete+compaction, and soft limits to find and remove the offending series.

AI-written summary · about

chapters (9)

  1. 0:00Introduction to cardinality
  2. 2:36Understanding Prometheus scale
  3. 5:23Alerting rule disaster
  4. 9:50Label value bloat
  5. 14:53Histogram configuration
  6. 20:34Metric name mismanagement
  7. 24:27Diagnostic techniques
  8. 28:30Solving label anomalies
  9. 32:08Q&A and mitigation strategies

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.