How SAP analyzes OpenTelemetry signals using Data Prepper
summary
SAP Cloud Logging uses Data Prepper to map OpenTelemetry to OpenSearch indices, running per-customer OpenSearch clusters on Kubernetes with a reverse proxy, mTLS, headless service for the Data Prepper peer forwarder, and operators to manage configs. Pipelines implement tail sampling, service-map and trace enrichment, and daily indices. Performance tests show roughly 30,000 logs/metrics/sec and ~5,000 trace spans/sec on two Data Prepper pods with a 2.9 GB heap; the peer forwarder and horizontal scaling for traces are current bottlenecks, so stability (circuit breaker at ~99.5% heap) beats peak throughput.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:37David Venable
- 1:03Agenda
- 1:38SAP Cloud Logging
- 3:02OpenTelemetry
- 3:45Data Prepper
- 5:17Data Prepper overview
- 7:21OpenTelemetry reference architecture
- 15:02Challenges with Data Prepper
- 15:31Component Architecture
- 17:05Kubernetes
- 18:30Object types
- 20:02Configuration
- 20:53Pipeline configuration
- 22:03Monitoring Data Prepper
- 25:40OpenTelemetry Metrics Explorer
- 26:31Data Prepper Performance
- 29:46Challenges Learnings
- 33:33Collaboration
related talks
26mUsing Data Prepper for Observability Ingest
29mOpenTelemetry With OpenSearch: Use-Cases, Status and Roadmap
42mIngestion at Scale: Open Source Pipelines for Getting Data Into Open...
24mIngesting Observability Data: OpenTelemetry Collector or Data Prepper?
24mUnlocking Advanced Processing Capabilities with OpenTelemetry
27mIngesting 6.5 Tb of Telemetry Data Daily Through Open Telemetry Protocol and Collectors
46mOperational Logs and Event Management for HPC S...
24mTo Make an SLO, First You Must Invent the Universe: OpenTelemetry Collection From Scra...