A Practical Blueprint for Evolving Observability Schemas
summary
A practical blueprint for evolving telemetry schemas that Shopify runs in production: a ClickHouse schema manager reads versioned YAML specs from GCS, reconciles logical to physical schemas, and applies non‑mutating changes (add columns, aliases) while rejecting expensive type rewrites. Pipeline uses OTLP JSON/Kafka, custom Go collectors, Prometheus/Thanos and a query layer that maps new and old paths. Promoted columns (~100) plus typed maps keep queries fast; unused mappings retire after 30 days.
AI-written summary · about
chapters (12)
- 0:00Observability schemas intro
- 1:03Why schemas matter
- 3:15Semantic convention changes
- 7:24Query-time schema handling
- 8:50Ingestion-time schema upgrades
- 10:14Schema management options
- 10:38Shopify’s ingestion schemas
- 15:17Logical vs physical schemas
- 17:02Schema versioning and CI
- 20:30Schema evolution blueprint
- 22:14Standardization and future
- 23:19Q&A
related talks
34mThe State of SQL-Based Observability
26mO11y in One: Exploring ClickHouse as a Unified Telemetry Database
32mOpenTelemetry Semantic Conventions and How to Avoid Broken Observability
21mInnovating on Top of Open Source Observability - Pushing the Boundaries o...
27mFast DIY O11y with OpenSrc Analytics & Visualization
33mObservability Pipeline Query Languages: Present and the Future
25mWhy We Built a Custom Exporter to Replace Our OTel Pipeline
23mSchema-First Application Telemetry