The Invisible Tax: How Data Format Conversions Drive up Telemetry...
summary
Measurements show OpenTelemetry SDKs (Go and Rust) spend most CPU on format hops — 80% spent on SDK conversion in Rust versus less in Go — and collectors add two expensive conversions per hop. A Rust Arrow engine using OTAP processes 10,000 records/sec with 512KB batches and Zstandard, yielding 40% less network bytes and lowering transform CPU (OTLP transform ~4.5% CPU vs OTAP ~1.7%) by using zero‑copy Arrow record batches. Recommendations: align on Arrow/OTAP to cut the transform tax.
AI-written summary · about
chapters (15)
- 0:00Welcome
- 1:26What is telemetry pipeline
- 2:28Format conversion costs in SDKs
- 4:36Rust and Go SDK overhead
- 5:40Avoiding intermediate conversions
- 7:02Telemetry after the SDK
- 7:25OpenTelemetry Arrow project
- 8:07OTAP reduces conversion costs
- 11:17Network bytes to memory formats
- 13:44Performance test setup
- 15:49Conversion cost results
- 17:14Telemetry pipeline recommendations
- 18:39Universal telemetry format fix
- 19:43OpenTelemetry Arrow as universal format
- 20:06Q&A
related talks
32mThe Invisible Tax: How Data Format Conversions Drive up Telemetry Pipeline Costs
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
36mObservability For You and Me with OpenTelemetry
45mSimplifying Instrumentation in complex systems at scale with Open Telemetry
27mBeyond OTLP: Unlocking the Potential of OS-native Tracing
15mTuning OTel Collector Performance Through Profiling
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
20mObservability and Open Telemetry