From Fragmented Logs to Unified Insights: Scaling O11y
summary
Dropbox moves from SSHing to a production logging pipeline built on Grafana Loki, ingesting about 10 GB/s and storing ~10 PB for 30 days across ~1,000 tenants. Promtail agents push logs, object storage (internal S3 replacement) cuts cost so retention rises from one week to four, and a custom query-auth proxy plus per-tenant rate limits and a memberlist replacing etcd ring solve scale and contention. WAL is disabled to favor availability, multihoming provides redundancy, and the deployment meets low-latency query and cost goals.
AI-written summary · about
chapters (11)
- 0:00Introduction and Agenda
- 0:28Background and Context
- 1:20Defining the Problem
- 3:05Requirements and Challenges
- 6:22Evaluation Metrics
- 7:53Logging Solutions Compared
- 9:50Why Grafana Loki?
- 11:56How Loki Works
- 18:37Loki Deployment at Dropbox
- 23:51Scaling Challenges and Solutions
- 27:46Conclusion
related talks
28mFrom Chaos To Clarity: Scaling Observability at Dropbox With Centralized Logging Solution
40mLarge-Scale Logging Made Easy
46mConquering Distributed Cloud Logging Without Breaking the Bank
30mData @Scale Seattle- Sundaram Narayanan
53mGetting started with Grafana Loki, a modern logs database
41mScaling Observability: From Chaos to Clarity with Open Source Tools
21mLogging at scale using Graylog - Billion+ messages, 100K req/sec
40mObservability that scales: Analyze data in stream and query remotely