Managing and Scaling Real Time Data Pipelines using Go
summary
Riot runs Go-based agents and collectors that push millions of player events daily through HTTP+JSON contracts into Kafka, Elasticsearch and Grafana; the system uses sidecars, batching and backpressure to stay resilient and horizontally scale. Core claims: Go pipelines hit 75k values/sec on a laptop, channels simplify conveyor-belt stages but mutexes outperform channels for high-read caches, and sync.WaitGroup plus closing channels yield graceful shutdown. Design accepts dropping stale metrics to keep alerts fresh and the system works in production.
AI-written summary · about
chapters (8)
- 0:00Start
- 9:16THE PIPELINE
- 10:27DESIGN GOALS
- 13:25METRIC AGENT
- 13:55COLLECTOR
- 20:39ORCHESTRATION: ORDERING
- 22:59API CLIENT: BATCHING (EASIER)
- 24:08API CLIENT: DRAINING
related talks
05mGo for Real Time Streaming Architectures
40mReal-time Customer Engagement in Gaming Using Kafka and Flink
06mHow We Achieved 100x Scale
30mFrom Service To Platform: A Ranking System in Go
53mGo at Mainframe Scale
30mHigh Performance Systems in Go
31mHow Go is Making us Faster
24mA Gentle Introduction to Building Data-Intensive Applications