Operating Opensearch - The Kubernetes Way
summary
Shows how OpenSearch runs on Kubernetes using the OpenSearch Operator to declaratively create stateful nodes, dashboards and TLS, with logs collected by Fluent Bit as a DaemonSet. High-volume growth causes Fluent Bit to crash, so a hub-and-spoke design inserts Kafka as a durable buffer; a custom worker replaces the Kafka-to-OpenSearch Fluent Bit consumer to enable complex filtering. Production logs go direct for low latency, non‑prod use Kafka; solution works but adds queuing latency and operational complexity.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:22Run OpenSearch on Kubernetes
- 4:33Kubernetes extensibility
- 6:38Kubernetes operators
- 9:50OpenSearch Kubernetes architecture
- 15:31Scaling multiple clusters
- 19:26Log volume growth challenges
- 21:35Kafka for log buffering
- 23:22Latency optimization
- 29:44Operator capabilities
- 33:16Deployment challenges
- 35:04Monitoring logging pipeline
- 36:35Q&A
related talks
39mOperating OpenSearch at Scale With Kubernetes Operators: Lessons Learned and Community Insights
52mOperating OpenSearch at Scale: Lessons From Managing 10,000+ Clusters Across...
23mCreating the OpenSearch Kubernetes Operator
42mOpenSearch in Kubernetes: Helm Charts and Operator
14mKeynote: OpenSearch on Kubernetes at Atlassian
36mAutoscaling Search and Analytics in Kubernetes: The OpenSearch Operator Advantage
09mIntroducing Opensearch in existing multisource enterprise App
40mRunning Solr within Kubernetes at Scale