Scale observability and secure data in motion with Elastic and Confluent
summary
Confluent Cloud plus Elastic Stack moves real-time telemetry (Kafka, Logstash, Elasticsearch, Kibana, APM) at massive scale — one customer streams 37 terabytes a day from four clouds across 20 regions with hundreds of apps, three Kubernetes clusters and 8,000 VMs. Architecture decouples ingestion from processing, enables prioritization and SLA tiers, and uses Schema Registry with the Elastic Common Schema and OpenTelemetry to enforce a single OTel Common Schema for logs, traces, metrics and security.
AI-written summary · about
chapters (11)
- 0:00Intro
- 3:24Your business needs access to data in motion
- 4:32Every business is becoming software
- 6:10Apache Kafka is an event streaming platfo
- 8:46Elastic and Confluent joint solution architecture
- 10:52Confluent + Elastic observability pipelines
- 11:42Lifecycle of Logs
- 13:35Financial services architecture
- 14:58How Elastic + Confluent combat fraud
- 16:32Different schemas: Complex and slow analysis
- 17:20Common schemas: Improve visibility with root cause analysis
related talks
52mGrowing to full-suite observability with Elasticsearch & the ELK Stack (PRT267)
50mIntuit Journey To Unified Observability at Scale: Challenges, Be...
23mWhen Kafka Goes Cloud Native: Observability That Actually Works!
41mScaling Telemetry Systems with Streaming
40mObservability that scales: Analyze data in stream and query remotely
35mFrom Chaos To Clarity: Centralizing Observability With OpenSearch at Scale
13mNext-Gen Observability: reduce MTTR
30mRunning Highly Scalable Tracing, Logging, Monitoring & Metric System in Production