Breaking Free from Vendor Lock-In: Nubank DIY Observability Success
summary
Nubank builds an in-house logging pipeline that ingests 1.5 petabytes and 1.5 trillion log lines per day, peaks at 45 GB/s, and reduces vendor costs by ~50% while running on a five-engineer team. Collection uses Fluent Bit at node level, transport uses S3 + SQS microbatching with 1–2 minute windows and KEDA autoscaling, storage uses Parquet + Trino on S3 with continuous compaction, 45-day retention, and near-real-time queries at 2–30 minute latency.
AI-written summary · about
chapters (18)
- 0:00Nubank observability challenge
- 3:32Why build our own platform
- 4:57Key log platform requirements
- 6:02Log ingestion with Fluent Bit
- 7:25Microbatching for cost savings
- 10:40Dynamic ingestion scaling
- 12:04Efficient log storage and querying
- 13:52Parquet for cost-effective storage
- 14:56Real-time Parquet compaction
- 16:00Querying logs with Trino
- 17:04UX with Grafana and plugins
- 17:46Domain-specific log search
- 18:30Architecture overview and results
- 19:32Q&A: OpenTelemetry and metrics
- 20:34Q&A: Accumulator buffer handling
- 22:00Q&A: Real-time search latency
- 23:29Q&A: Cost per customer
- 24:55Q&A: Data retention strategy
related talks
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
40mObservability that scales: Analyze data in stream and query remotely
33mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenges
30mData @Scale Seattle- Sundaram Narayanan
41mScaling Observability: From Chaos to Clarity with Open Source Tools
17mThe Observability Money Heist - a Fintech's Tale of...
50mIntuit Journey To Unified Observability at Scale: Challenges, Be...
28mFrom Chaos To Clarity: Scaling Observability at Dropbox With Centralized Logging Solution