First Principles of Observability Systems
summary
Argument: index the right amount, not everything or nothing. Examples: Loki stores gzipped log chunks in object store and keeps only boltdb metadata so queries get slow at scale; Uber pairs Kafka with ClickHouse and uses dynamic indexing to index only frequently queried fields, trading write throughput for targeted reads; some orgs use per-window downsampled tables (15m/30m/1h) and Bloom filters to avoid full inverted indexes; Netflix stores nothing by default and only persists user-requested traces, cutting storage cost. ClickHouse wins for codecs (lz4, zstd), columnar files, tiered storage and data-skipping indexes but needs ops for horizontal scaling.
AI-written summary · about
chapters (17)
- 0:00Observability systems intro
- 0:41Observability challenges evolution
- 2:08Observability pipeline components
- 3:12Why ELK stack abandoned
- 5:24Loki’s log observability
- 7:11ClickHouse as data store
- 8:59Compression and indexing trade-offs
- 10:41Minimal indexing projects
- 11:25Netflix log storage strategy
- 12:31Efficient indexing principles
- 13:12Delta encoding in storage
- 13:55Data correlation importance
- 14:39Why ClickHouse popular
- 16:24Multi-tenancy compatibility
- 17:06OpenTelemetry standardization
- 17:48Future query standards
- 18:54Q&A
related talks
34mThe State of SQL-Based Observability
24mWhat Observability Can Learn From BI
10mWhy your observability is costing you too much?
30mObservability is Too Expensive - Chris Cooney
29mWide Event Analytics
26mO11y in One: Exploring ClickHouse as a Unified Telemetry Database
35mDriving down the cost of observability
52mObservability: it's not just an ops thing