Taming Tenancy, Cost and Architecture at Collibra Through OpenTelemetry and Our...
summary
Costs skyrocket when giving each tenant its own container, so Collibra switches to multi-tenancy and saves money while keeping per-tenant attribution by assigning UUID tenant IDs and enforcing single-line JSON logs that contain tenant info. A custom OpenTelemetry backbone uses daemonset collectors for traces metrics and logs, funnels signals into Google Pub/Sub, enriches events from master data to promote JSON fields into OpenTelemetry attributes, and routes to backends and BigQuery with daily batch loads to cut storage cost; cost-by-C4 architecture labels is calculated and operational.
AI-written summary · about
chapters (8)
- 0:00Collibra’s telemetry journey
- 2:51Monolith to multi-tenant challenges
- 7:28Golden attributes for tenancy
- 13:11Enriching telemetry with master data
- 16:44Observability and cost insights
- 19:11Calculating telemetry costs
- 21:21Semantic conventions and AI
- 23:31Key takeaways
related talks
32mDesign Observability Solution for Multi-Tenancy Environment
40mMulti-tenant Logging with Opentelemetry Collector- 40mMulti-tenant Logging with Opentelemetry Collector
29mUnlocking Business Insights with OpenTelemetry
34mPanel: OpenTelemetry: Bridging Platform and Enablement
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
36mObservability For You and Me with OpenTelemetry
28mFrom Fragmented Logs to Unified Insights: Scaling O11y