The Challenges of Managing Open Source Infrastructure at Bloomberg
summary
Telemetry as a Service handles over 6M metrics/sec and ~120 TB of logs daily, so decisions focus on scale and failures: Kafka for ingestion, MetricTank as a MetricTank drop-in replacement fronting Cassandra for in‑memory caching and rollups, Spinnaker+Helm for automated kubernetes deployments, a custom Kubernetes scheduler to avoid node overload during memory‑heavy backfills, a partition‑reassignment wrapper (and LinkedIn Kafka tools PR) to throttle rebalance traffic, a custom Windows collector replacing collectd; these changes reduce timeouts, prevent ingest lag, and improve reliability while still prompting experiments with ScyllaDB.
AI-written summary · about
chapters (7)
- 0:00Start
- 0:49Bloomberg Overview
- 1:43Telemetry at Bloomberg
- 13:47Growing Pains: Deploying Containerized Applications
- 15:25Growing Pains: Scheduling Kubernetes Pods
- 20:44Growing Pains: Katka Partition Reassignments
- 25:13Growing Pains: Summary
related talks
41mScaling Telemetry Systems with Streaming
26mScaling OpenTelemetry for Modern Workloads: From Mobile To Ma...
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
49mAdding OpenTelemetry to Production Apps Lessons Learned
41mScaling Observability: From Chaos to Clarity with Open Source Tools
29mAI Driven Observability based on Open Source
05mTelemetry Pipelines: Your Recipe for Success
30mRunning Highly Scalable Tracing, Logging, Monitoring & Metric System in Production