The Power of Consolidation: A Unified Stack for Business Intelligence, Securi...
summary
A reproducible three‑container stack demonstrates a unified backend for observability, security, and BI: ClickHouse as the columnar warehouse, an OpenTelemetry Collector pipeline, and Grafana for ad‑hoc queries, dashboards and alerts, with MinIO mocking S3 and an Iceberg+Parquet integration (Project Antalia) for long‑term retention. Real customers hit orders of 50 PB total historically and up to ~2 PB/hour ingestion; the demo shows link‑click analytics, flame‑graph traces, bot detection, and performant queries via ClickHouse’s MergeTree compaction.
AI-written summary · about
chapters (18)
- 0:00Introduction
- 1:02Disparate data systems problem
- 2:51Wide events challenges
- 4:40Security observability vs BI
- 7:11Unified data architecture challenges
- 9:57Investigative loop and time queries
- 12:47People challenges in unified solutions
- 13:29Desired stack characteristics
- 15:33Architecture overview
- 17:58ClickHouse and Grafana demo
- 22:13Open source alternatives
- 23:35Scaling for availability
- 24:59ClickHouse performance at scale
- 26:44Scaling with S3 and Iceberg
- 28:09Discoverability and integrations
- 28:49Self-service templates
- 30:12Final thoughts on tools
- 31:38Q&A
related talks
26mO11y in One: Exploring ClickHouse as a Unified Telemetry Database
24mHow We Built ClickStack - an open source O11y stack
17mBuilding Real-Time Data Lake with ClickHouse
28mColumnstore Revolution
34mCatalogs, Iceberg, and Delta: Building the Foundation for Lakehouse Analytics
27mFast DIY O11y with OpenSrc Analytics & Visualization
47mDistributed high-load BI engine for Google Data Studio and Microsoft Power BI
24mWhat Observability Can Learn From BI