That's Not a Lot of Data; Leveraging Cloud-native Technologies to Collect, Stor...
summary
Cloud-native observability stack is used to collect and analyze financial timeseries at production scale: Prometheus: 100M+ active series on a single instance, Cortex: 600M+ series per cluster, and a customer running 3 billion active series. Grafana visualizes and correlates metrics, Loki ingests high-throughput logs with tens of GB/s query throughput, and Tempo stores 2.2M traces/sec with p99 <2.5s; Robinhood scaled a Prometheus deployment from ~100M to ~700M series during the GameStop surge.
AI-written summary · about
chapters (9)
- 0:00Intro
- 1:26History of Data
- 3:11Observability
- 6:39Prometheus
- 10:13Loki
- 13:03Tempo
- 15:09Grafana
- 17:13Business Model
- 18:19Conclusion
related talks
16mKeynote: Prometheus 2.0 – The Next Scale of Cloud Native Monitoring
20mHow We are Dealing with Metrics at Scale on GitLab.com
56mUnwiring High Cardinality
32mPrometheus Deep Dive
12mUnlocking the Secrets of Client Data
13mLightning Talk: A Drop-in System To Accelerate Metrics Observability by 100x Us...
16mShrinking the Observability Bill
23mAutomatically Making Dashboards Load 100X Faster