Tracing Real-Time Distributed Systems
Talkat SREcon19 Europe/Middle East/Africa · on Thursday, 2019-10-03· at 14:45· Track 1: The Liffey B·
summary
Bloomberg traces real-time trading pipelines by embedding a span-based tracer in the messaging library, sending data to a custom backend while reusing Jaeger’s UI, and handling ~100 billion market ticks/day and ~500 million spans/day. Challenges include fan‑out/fan‑in and conflation (80% message discard in hot backups), chunked initial loads, and async “follows-from” edges; solutions include creating dispatch spans for message parts, semantic tags as lightweight checkpoints, deferred sampling decisions from buffered windows, and extracting SLO metrics and alerts from derived latency signals.
AI-written summary · about
chapters (22)
- 0:00Intro
- 0:32Overview
- 2:27What is the Bloomberg Terminal?
- 3:01About Bloomberg
- 4:48Example - A Search Engine
- 5:45(Possible) Architecture View
- 6:58Trace Concepts. Span Model
- 8:00Tracing Architecture (One Interpretation)
- 10:35Using Trace • Triage Debugging
- 11:57Tracing Model & Implementation
- 15:34A Trading System
- 17:15Architecture - Simple View
- 18:34System Characteristics
- 19:52How we traced it?
- 21:23What does it look like?
- 22:01The interesting bit
- 23:48What we do with it?
- 26:40Message Fan-Out (broadcast)
- 28:05#2 Splitting Messages
- 30:15Message Conflation
- 33:55What's Next?
- 35:22We are hiring!
related talks
49mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
57mDistributed Tracing: How the Pros Debug Concurrent and Distributed Systems
22mDistributed Tracing
33mDistributed Tracing - All The Warning Signs Were Out There!
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch
26mDistributed Tracing: All the Warning Signs Were Out There!!!
34mReduce Complexity with Distributing Tracing
11mCognitive and Self-Adaptive System for Effective Distributed-Tracing...