The Observability Data Lake, 1 year on
summary
Netflix runs an Observability Data Lake in production ingesting about 160 billion traces (≈1 trillion spans) and three petabytes of logs per day, storing data on S3 parquet files with a custom byte-level index that’s ~16% of raw size. Queries fan out to Lambda, stream partial results, and deliver most responses in ~2–3 seconds with hotspots under 10-second latency for live dashboards; result: ~66% lower cost than Cassandra+Elasticsearch and production-readiness for tracing.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:23Challenges
- 3:27User Scale
- 4:26Cost
- 5:50Solutions
- 7:02Namespaces
- 8:58Service Catalog
- 11:22Cost Control
- 15:00New Use Cases
- 18:27Latency
- 24:31Vendors
related talks
59mAdvanced Observability Strategies for Media Workflows at Netflix
49mData Platform Architecture, Evolution, and Philosophy
31mMastering Near-Real-Time Telemetry and Big Data: Invaluable Superpowers for Ordinary SREs
30mRaw Data to Business Insights: What You Need to Build a Modern Data Lake
13mNext-Gen Observability: reduce MTTR
39mLarge scale streaming analytics
33mNetdata: Open Source, Distributed Observability Pipeline – Journey and Challenges
15mThree key considerations for deploying best-of-breed observability