Why We Built a Custom Exporter to Replace Our OTel Pipeline
summary
ClickHouse-to-ClickHouse native transfers replace the OTEL JSON pipeline, copying system tables byte-for-byte with a Go client to avoid parsing, batching overhead and log-rotation data loss; dynamic schema replication plus a MergeTable engine view makes hundreds of per-pod tables appear as one; the custom exporter CSX (system tables exporter) ships error/trace/part logs that never hit stdout; result: from ~8,000 CPU cores for 2M logs/s with OTEL to 70 cores for 37M logs/s, proven on a 100PB, 500 trillion-row platform.
AI-written summary · about
chapters (9)
- 0:00Introduction and Audience Engagement
- 0:20Introduction to ClickHouse
- 2:17Challenges with Open Telemetry
- 2:51The Log House Architecture
- 6:20Inefficiencies and Data Loss
- 8:48Introducing CSX: System Tables Exporter
- 10:43Dynamic Schema Generation and Merge Table Engine
- 12:59CSX vs Traditional Metrics Systems
- 19:52Q&A Session
related talks
29mA Practical Blueprint for Evolving Observability Schemas
27mFast DIY O11y with OpenSrc Analytics & Visualization
21mBuilding Composable OTel Pipelines: CI/CD, Testing, Team-First, and Sca...
28mExperiments in Backing Prometheus with Clickhouse
37mFinding Bottlenecks in ClickHouse: Profiling, Heatmaps & Flamegraphs
26mDecoding OpenTelemetry(OTel) Collector: A Deep Dive into Architecture
15mLogs Parsing Using OpenTelemetry and SigNoz
30mCheap DIY Monitoring with OSS Analytics & Visualization