Lightning Talk: Scrape Trolley Dilemma
summary
Prometheus and OpenTelemetry collectors hit memory limits when a single scrape brings high cardinality series and the common outcome is either a crash loop or ad hoc drops. Current push receivers instead shift burden to clients by dropping or backpressuring. Proposed specifics: implement an optional flag that automatically evicts selected series under memory pressure, try early compaction and dynamic scrape-rate throttling, and add a metric criticality attribute so autoscaling or alerting series are preserved. Naive prototype exists conceptually and the plan is to iterate with user feedback.
AI-written summary · about
related talks
17mLightning Talk: Avoiding Chaotic Dysfunction: The ‘Don’t Do This’ Guide T...
11mLightning Talk: Beyond Billions: Operating Thanos, Prometheus & OpenTelemetry...
05mLightning Talk: Transactional Remote-write and Why You Should Care
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
12mLightning Talk: A Drop-in System To Accelerate Metrics Observability by 100x Us...
12mLightning Talk: With Great Traces Comes Great Costs: How to Reduce That Bill?
11mLightning Talk: “Metrics That Lie”: Understanding OpenTelemetry’s Cardinality Capping...
13mLightning Talk: A Drop-in System To Accelerate Metrics Observability by 100x Us...