Lightning Talk: Federated Search Over Distributed Observability Data
summary
Operates on petabytes of observability data per day and billions of transactions, keeping raw and derived logs, traces and metrics distributed to avoid network and storage costs. Implements federated search that processes where data lives, centralises only metadata, and stitches results across stores using OpenTelemetry, Apache Arrow (with Flight and DataFusion) and Apache Presto plus Iceberg for flexible metadata; prototype reduces egress and compute while preserving correlation and query performance.
AI-written summary · about
chapters (8)
- 0:00Federated search intro
- 1:26Distributed observability challenges
- 3:15Federated search solutions
- 5:41Apache Arrow and Flight
- 7:04Gaps in federated search
- 8:08Presto for observability
- 9:12Hive Metastore limitations
- 10:15Future tools and directions
related talks
50mIntuit Journey To Unified Observability at Scale: Challenges, Be...
14mKey Considerations For Success When Deploying A Solution For Best-of-breed Observability At Scale
24mWhat Observability Can Learn From BI
35mFrom Chaos To Clarity: Centralizing Observability With OpenSearch at Scale
21mKubernetes Under the Magnifying Glass: Observability With OpenSearch
06mSponsored Keynote: OpenSearch - See Everything: Open Observability for Agentic AI
11mLightning Talk: OTel Me How to Build a Data Pipeline for Observability
05mLightning Talk: Retire the Sidecar: Why Observability Needs To Ride up Front