TTChat’s Story: Connect Metrics, Logs and Traces with EBPF
summary
Started with client SDKs sending ~200TB/day, then shifts to eBPF agent for auto-instrumentation to avoid code changes; uses OpenTelemetry Collector plus a custom span-metrics connector that turns spans into metrics and logs, with standard span attributes and middleware to reduce config drift. Stores raw eBPF data in ClickHouse for TCP-sequence based chase reconstruction, samples error/long spans, and reports over 600,000 spans/sec per cluster; dropping eBPF traces after transform saves ~30% CPU, ~70% memory.
AI-written summary · about
chapters (12)
- 0:00TTChat observability journey
- 1:04Metrics and logs from traces
- 3:09Correlating signals without code changes
- 6:20Span-to-metrics overhead
- 8:10Standardizing spans for metrics
- 10:16Prometheus high-cardinality protection
- 11:44Prometheus native histograms
- 13:09eBPF data collection without code changes
- 15:13Linking eBPF spans without trace IDs
- 16:38eBPF storage and resource costs
- 17:42Final architecture and plans
- 19:31Q&A
related talks
35mAn Exemplary Path: Leveraging EBPFs and OpenTelemetry To Auto-instrument for Exemplars
26mDistributed Tracing: All the Warning Signs Were Out There!!!
31mZero instrumentation observability with eBPF
33mDistributed Tracing - All The Warning Signs Were Out There!
23mEnhancing Asynchronous Communication Observability with OpenTelemetry
34mNetwork Observability with eBPF and OpenTelemetry
31mBridging Dev and Ops with eBPF: Extending Observability Upwards and Downwards
22mUnified Observability