Refining Systems Data without Losing Fidelity
Talkat SREcon19 Europe/Middle East/Africa · on Thursday, 2019-10-03· at 14:00· Track 1: The Liffey B·
summary
Observability at production scale is unaffordable if every event is stored: a 99.95% SLO produces ~2,000× more “good” events than failures. The concrete plan is reduce reuse recycle: trim and structure logs, apply per-key adaptive sampling (e.g. roll a die, sample 1/6 or 1/10k, adjust by trailing-minute volume), buffer full traces briefly (Lightstep/Honeycomb style) to decide retention, and keep all errors and slow traces while weighting retained samples by their sample rate for accurate inverse-quantiles. OpenTelemetry still needs sample-rate propagation.
AI-written summary · about
chapters (30)
- 0:00Intro
- 0:17Complex systems are hard to manage.
- 1:11User experiences.
- 2:34User experiences marbles.
- 5:37without breaking the bank?
- 6:04Three strategies for taming the spew.
- 6:15Reduce. Reuse. Recycle.
- 6:32Store less data.
- 7:15Stop writing read-never data.
- 7:34First, structure your data.
- 8:12One event per transaction.
- 9:12Often, trimming isn't enough.
- 9:32Sample your data.
- 9:45Statistics to the rescue!
- 10:05Count 1/N events.
- 11:01Count traces together.
- 12:18Don't be afraid of sample rates.
- 13:01Don't believe me? Ask a data scientist.
- 15:17Aggregate data.
- 15:29Aggregation destroys cardinality.
- 17:30Temporal correlation is weak.
- 18:25Math on quantiles is misleading.
- 19:44Aggregation is a last resort.
- 20:03How can sampling be cheap enough?
- 20:47Systems scale with load.
- 22:11Reconcile using the sample rate.
- 24:09How can we save the relevant events?
- 24:43Different key, different probability.
- 25:26Retain errors & slow queries.
- 29:29Metrics and events can be friends!
related talks
33mReal-World Sampling – Lessons Learned...
06mObservability Diet: Your 5-Step Plan To Trim the Data Fat
53m.NET Rocks Live
38mThe Murphy's Laws of Observability
39mShifting Left on Production Excellence with Observability
32mRules of Observability: The Data Edition
35m99.99% of Your Traces Are (Probably) Trash
25mMaking Observability Work at the Edge