Intuit Journey To Unified Observability at Scale: Challenges, Be...
summary
Intuit-level scale observability unifies 320+ Kubernetes clusters, ~8,000 services, 2 billion active time series, 10M trace spans/sec and 1.2PB logs/day peak into a federated metadata/catalog layer instead of centralizing raw data. Stack uses OpenTelemetry ingestion, Parquet + Iceberg on S3, Trino federated SQL, ClickHouse for golden signals, Arrow/DataFusion or DuckDB for in‑stream queries, and a lightweight Neaflow stream processor. Resulting system yields ~70–80% cost savings and cuts MTTD from 23 to 8 minutes and MTTR from ~100 to ~30 minutes while adopting selective LLMs for SQL generation and trace summarization.
AI-written summary · about
chapters (12)
- 0:00Unified observability journey
- 3:52Scaling observability challenges
- 9:09Unified observability goals
- 15:52OpenTelemetry architecture
- 23:20AI-driven query generation
- 28:22Alerting with anomaly detection
- 33:00Logs pillar challenges
- 35:49Metrics scaling efficiency
- 36:54Traces and events optimizations
- 38:39Next steps for profiling
- 41:03Agentic AI for recovery
- 46:10Q&A
related talks
13mNext-Gen Observability: reduce MTTR
41mCustomer Centric Observability at Scale Leveraging AIOps & OpenTelemetry
34mThe State of SQL-Based Observability
12mMastering Observability in Distributed Systems
24mWhat Observability Can Learn From BI
23mAnalyzing Operational Data at Scale Using ML at Intuit
28mScaling Observability for Real-Time Personalization Engines
23mImplementation of Unified Observability at Scale From Scratch