From Overwhelmed to Optimized: How We Cut Envoy Logging Costs by 65% While Boosting Throughput 60%
summary
Envoy logging at Databricks cuts 65% logging cost reduction and delivers a 60% throughput boost by replacing expensive cell extractors with direct/header/metadata extractors, adding low-cost access-log filters and a three-tier logging policy (100% essentials, error-only tier, 10% deep-diagnostics sampling). Microbenchmarks quantify extractor cost in ncond, show CPU burn from 17% to single digits, move throughput from 800→1,800 QPS per vCPU, save ~$400k/year in compute, and keep 99.9% uptime while shrinking MTTR from hours to minutes.
AI-written summary · about
chapters (8)
- 0:00Envoy logging challenges
- 1:51Envoy architecture and logging
- 3:16Logging impact on throughput
- 5:23Why access logs matter
- 6:24Log extractor cost differences
- 9:35Optimizing with logging filters
- 11:42Three-tier logging strategy
- 12:47Optimization results
related talks
31mMy Logs Cost More Than My Services
26mIntegrating Stateful Load Balancing At Scale In Envoy
35mDriving down the cost of observability
35mHigh Throughput with Low Resource Usage: A Logging Journey
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
27mEnvoy - Overview and Update: Senior Envoy Maintainer
40mLarge-Scale Logging Made Easy