Cache Me If You Can: How Grafana Labs Scaled Up Their...
summary
Grafana Loki adds large local SSD-backed memcached (mcached with extstore) clusters—33 nodes with four 375GB local SSDs each—to reach ≈50 TB memcached cache and ~0.5 Tbps aggregate network. That reduces object-storage requests by 65%, raises daily hit rates to ~95%, cuts un-fetched evictions dramatically, yields an unexpected 2% TCO improvement, and keeps query performance acceptable despite p50 cache latency sometimes rising toward 50 ms.
AI-written summary · about
chapters (7)
- 0:00Introduction to the problem
- 1:00Understanding Loki internals
- 4:40Reliability and scaling issues
- 7:24Caching strategies and needs
- 12:45Implementing Memcached on SSDs
- 17:45Operational results
- 21:28Trade-offs and conclusion
related talks
19mKeep Storing and Carry On
25mGreenOps at Scale: The 50TB Memory Diet for a Petabyte Platform
23mHow to edge cache GraphQL APIs
37mCaching Deep Dive: The Ultimate Way To Speed up Your GraphQL API
23mScaling Memcache
15mAccelerate Application Response Times with True Cache
01h 04mHow do you store your data?
30mStateful Superpowers: Explore High Performa...