The Daemonset in the Details: Pitfalls of OpenTelemetry Collector Deployments
summary
Deploying OpenTelemetry collectors across hundreds of K8s clusters with ArgoCD and Helm reveals concrete failures: Helm rendering writes invalid CRD fields (spec.replicas) that the API server rejects, collector pods crash from a bad config key ("Trace" vs "traces"), and distribution-specific components break naive validation. The response is to build Go unit tests that render templates, unmarshal into the Otel CR, run the collector binary’s validate logic with an in-memory components provider, and execute terot test harnesses for OTL transforms — these tests catch the issues before cluster deploys.
AI-written summary · about
chapters (6)
- 0:00Introduction to OpenTelemetry
- 1:55Helm chart unit testing
- 7:28Validating configuration
- 10:48Deep testing configurations
- 17:56Transforming telemetry
- 20:47DSL implementation and wrap-up
related talks
34mKubernetes Observability with OpenTelemetry
24mWhen Things Go Sideways: Troubleshooting the OTel Operator
28mImplementing Observability in Kubernetes
26mTurn the Volume Down on Noisy Neighbors!
40mMulti-tenant Logging with Opentelemetry Collector
36mShould I use OTel (collectors), or is Prometheus good enough?
27mTop 5 Things to Know Running OpenTelemetry on Kubernetes
46mObservability Ecosystem in Kubernetes