Drowning on Metrics or why Jack could have fit
summary
Lidl Plus grows from 2 stores to 13,000 and 100k to 90M users, so capacity planning uses ticket counts to infer country users and runs component and end‑to‑end load tests per rollout; the stack moves from Firebase/Dynatrace to OpenTelemetry with Grafana, Kafka and Kubernetes. Observability faces high‑cardinality OOMs, so teams build cardinality dashboards, load‑test collectors, drop high‑cardinality labels, downsample spans and enforce a platform team. Load test tools include k6 and JMeter; goal: cut cloud cost by ~30%.
AI-written summary · about
chapters (18)
- 0:00Introduction and setup
- 1:14From performance to storytelling
- 3:05Project challenges and goals
- 5:33Scaling infrastructure and microservices
- 8:24Predicting user growth with statistics
- 10:11Load testing and efficiency
- 12:20Defining SLOs and SLAs
- 14:09Peak traffic and budget balance
- 16:38Moving to Kubernetes for cost savings
- 20:12Observability challenges after migration
- 22:42Standardizing with OpenTelemetry
- 25:14High cardinality and metric explosion
- 27:01Reducing observability costs
- 29:09Testing observability under load
- 31:03Quality over quantity in metrics
- 32:07Scaling the team with a platform team
- 35:08Reducing costs and improving resilience
- 36:10Q&A session
related talks
11mLightning Talk: “Metrics That Lie”: Understanding OpenTelemetry’s Cardinality Capping...
44mGrafana Labs Customers: What We've Learned Building Observability at Massive Scale
34mMetrics driven development
41mResilient Observability at the Retail Edge: A Lightweight, Scalable, and...
41mScaling Observability: From Chaos to Clarity with Open Source Tools
27mTaking Control of Metrics Growth and Cardinality: Tips for Maximizing Your Observability
19mUser-centered observability: load testing, real user monitoring & synthetics
29mAI Driven Observability based on Open Source