Why it Takes a Team of Engineers to Process Events in RT
summary
Segment separates control plane and data plane, shipping a control store: per-host read-only SQLite populated by a reflector so services read local configs even if upstream services fail. Deduplication sits before destinations using Kafka and local RocksDB; output messages carry the upstream offset so recovery replays only missing IDs. High-throughput analytics move to ClickHouse with materialized views behind a query gateway per-database to control queries. Core services and tooling are written in Go, enabling migrations, low-overhead metrics, and the scale and reliability gains described.
AI-written summary · about
chapters (15)
- 0:00Intro
- 2:48Microservice Availability
- 4:23Availability Math
- 5:23Goodbye Microservices
- 6:47Conceptual Architecture
- 7:31Architecture In Practice
- 8:36Actual footage of us brainstorming
- 8:48Current Architecture
- 11:52Consistency
- 12:35Deduplication
- 15:39Key Insight
- 16:27Click House
- 17:30Query Gateway Pattern
- 20:12segmentio/queues
- 22:20segmentio/stats
related talks
08mReal-Time Data Systems: Speed, Scale, and Resilience
24mReal-Time Streaming Segmentation on Azure
57mLessons in Building a System that Processes More than 70 Billion Events Daily
35mReal-time analytics at IoT scale
30mData Platform Infrastructure at Rivian: A Deep Dive
28mManaging and Scaling Real Time Data Pipelines using Go
26mKeynote: 10 Million Spans Per Second: Lessons From Scaling OpenTelemetry at Reddit
25mAccelerating Data Processing and Consumption for Building Modern Apps at the Edge