Cortex 101: Horizontally Scalable Long Term Storage for Prometheus
Talkat KubeCon + CloudNativeCon North America 2019 · on Tuesday, 2019-11-19· at 11:50· Room 15AB - San Diego Convention Center, Mezzanine Level·
summary
Cortex implements horizontally scalable, highly available, multi-tenant long‑term storage for Prometheus by vendoring Prometheus code and PromQL, accepting remote_write and doing HA deduplication via a distributor → ingesters pipeline with consistent hashing. Samples compress with Gorilla-style encoding to about 1.37 bytes/sample and flush as chunks to object stores and indexes in DynamoDB + S3, BigTable + GCS, or Cassandra. Query front-end splits large ranges and caches queries; ruler evaluates and writes rules back to ingesters. Experimental features include blocks shipping and gossip to remove etcd/consul.
AI-written summary · about
chapters (17)
- 0:00Intro
- 0:19About me
- 1:20Kubernetes and Prometheus
- 2:41Why use Cortex?
- 3:03Practical Use Cases
- 3:59Who uses Cortex?
- 4:58How does Cortex do it?
- 5:57Cortex: Global View
- 6:36Cortex: Single Binary Mode
- 7:25Cortex: How to Scale?
- 9:36Cortex: Ingest Compression
- 10:43Cortex: Query
- 12:52Cortex: Ingest and Query
- 13:26Cortex: Global Rule Evaluation es
- 15:40Cortex: Long Term Storage
- 17:17What's New?
- 19:12Thanos: An Alternative
related talks
37mIntro: Cortex
16mCortex: Multi-tenant Scalable Prometheus
39mDeep Dive: Cortex
25mCortex: Ingesting and Querying 100s of Billions of Samples Per Day
20mCortex: Intro and Production Tips
22mCortex: Ingesting and Querying 100s of Billions of Samples Per Day
05mMigrating Prometheus data between different storage systems
43mChronix as Long Term Storage for Prometheus