From Chaos to Consistency: OTel Sidecars and Semantic Conventions in...
summary
Company deploys one OpenTelemetry Collector sidecar per component across ~2,000 Azure subscriptions and 1,700 VM scale sets to get a single pane of glass and UTC-aligned telemetry the fleet produces lower latency and no crashes so far but managing 1,200+ collectors remains hard. Iteration two replaces Helm and Bicep with Crossplane ring-based rollout and adds Weaver merge gates to enforce OpenTelemetry and company semantic conventions. Persistent problem is consistent sampling for long 5–10 minute operations and exploring tail-based sampling challenge solutions.
AI-written summary · about
