Cortex: Ingesting and Querying 100s of Billions of Samples Per Day
summary
Introduces Prometheus and then describes Cortex, a horizontally scalable distributed implementation designed to ingest and query hundreds of billions of samples per day. Covers Cortex’s architecture—including decoupled ingest and query paths that scale independently—traces the original design, the bottlenecks encountered, subsequent architectural changes, and current community work and future plans.
AI-written summary · about
chapters (19)
- 0:00Intro
- 1:04Prometheus
- 2:01Datamodel
- 3:50The Problem
- 4:25Requirements
- 5:02Solution: Project Frankenstein
- 5:57Frankenstein - Cortex
- 8:21Different ingest and query path
- 8:42Better indexing
- 10:10Index Caching
- 11:38Query Caching
- 12:28Query Parallelisation
- 13:32WIP
- 15:24Getting Started
- 15:56Simple Setup
- 16:29You're done!
- 17:27Easy to operate?
- 18:33Status tomorrow
- 19:36TSDB Inside Cortex
related talks
25mCortex: Ingesting and Querying 100s of Billions of Samples Per Day
37mIntro: Cortex
39mDeep Dive: Cortex
20mCortex: Intro and Production Tips
28mCortex 101: Horizontally Scalable Long Term Storage for Prometheus
16mCortex: Multi-tenant Scalable Prometheus
23mHow we converted our Cortex data to TSDB blocks
26mBetter Scalability & More Isolation? The Cortex “Shuffle Sharding” Story