Fast DIY O11y with OpenSrc Analytics & Visualization
summary
Builds a working observability pipeline using ClickHouse + Grafana, a 13-line Python producer that parses vmstat into JSON, and a JSONEachRow HTTP insert (curl) to stream data into a ClickHouse table. Live demo uses stress to push CPU to ~90% and dashboards update in seconds. Scale notes: prefer big blocks (~100k–1M rows), use Kafka + ClickHouse Kafka table + materialized view for many collectors, expect 90% compression on metrics, supports sharding (50 shards), cloud nodes up to ~70 TB, and paths to Iceberg for 100+ PB.
AI-written summary · about
chapters (8)
- 0:00Introduction and Welcome
- 0:20Speaker Background and Experience
- 1:07Introduction to ClickHouse
- 1:30Building an Observability System
- 4:26Setting Up ClickHouse and Grafana
- 13:03Live Demo: Monitoring System
- 16:14Scaling and Optimization Tips
- 26:38Q&A Session
related talks
30mCheap DIY Monitoring with OSS Analytics & Visualization
26mO11y in One: Exploring ClickHouse as a Unified Telemetry Database
23mThe Open Source Analytic Universe, Version 2022
24mHow We Built ClickStack - an open source O11y stack
17mUsing ClickHouse® Database to Power an Analytics and Customer Engagement Platform
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns
34mThe State of SQL-Based Observability
55mBuilding Fast ClickHouse Analytics for MySQL Transaction Data Using Kubernetes