The Invisible Tax: How Data Format Conversions Drive up Telemetry Pipeline Costs
summary
Telemetry conversion dominates CPU and memory costs across SDKs, collectors, and exporters; a prototype Arrow-based Data Flow Engine and a new OTAP (OpenTelemetry Arrow Protocol) replace repeated marshal/unmarshal with a near-zero-copy, columnar layout, cutting conversion work and GC. Measured results show roughly 2.5 million logs/sec on one core (about 20x OTLP), renames and filters become essentially free, and the prototype works but is early-stage and not production-ready.
AI-written summary · about
chapters (14)
- 0:00Telemetry pipeline costs
- 1:49Telemetry pipelines role
- 3:55Pipelines CPU usage
- 9:48Conversion costs in SDKs
- 13:58Data format conversion problem
- 15:22Apache Arrow for telemetry
- 17:08OTAP and Arrow collector
- 19:13Arrow engine performance
- 21:40Arrow vs other formats
- 23:26Arrow columnar bulk ops
- 24:28OpenTelemetry Arrow phases
- 26:14Key takeaways and future
- 27:39Get involved in project
- 28:00Q&A exporters statefulness
related talks
22mThe Invisible Tax: How Data Format Conversions Drive up Telemetry...
07mSponsored Keynote: Datadog - Every Byte Counts: How Protocol Design Shapes the Cost...
36mObservability For You and Me with OpenTelemetry
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
13mLightning Talk: Taming Metric Cardinality: Practical Cost Reduction Wit...
32mTaming the Cost of Telemetry: How Riot Games Reined In...
24mScaling Telemetry To Terabytes: Applying Google's Principles To OpenTelemetry...