Running Highly Scalable Tracing, Logging, Monitoring & Metric System in Production
summary
Claims a production setup that actually serves 252 million subscribers and handled 37 million new users in nine months while running its own data centers. It solves async context loss and sampling by using Zipkin/Jaeger with Kamon for context propagation and implements log streaming and indexing with Kafka + Logstash + Cassandra + Elasticsearch, plus Kibana/Prometheus/Grafana for visualization and alerting. The pipeline is built end to end, tuned for punctuate/partition issues in Kafka, and runs at scale in production.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 1:37GeoMoney App
- 3:09Environment
- 7:15Logging
- 8:42Log Streaming
- 10:38Synchronous Programming
- 12:26Log
- 13:13Programming Languages
- 14:18Log Stash
- 15:06Kibana
- 16:10Different Logs
- 17:09Tracing
- 18:51Open Tracing
- 19:40Sampling
- 21:02Mon
- 22:17Zipkin
- 23:16Jagger
- 24:36Monitoring
- 24:55Prometheus
- 25:48Prometheus Problems
- 26:45Kafka
related talks
41mScaling Telemetry Systems with Streaming
45mMastering Distributed Tracing: Insights from Building a Platform from Scratch
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
46mBuilding a scalable logging platform
34mEfficient Resource Management
45mTracing: A journey to tactical insights
24mData analytics with distributed tracing data
45mUsing time-series data to better understand our products & infrastructure