A Journey of Auditlogs from Kafka to ElasticSearch
summary
Built a centralized audit pipeline that writes Avro-compressed events to MSK Kafka, uses a Confluent connector to bulk index into OpenSearch and concurrently writes Avro files to S3 for 3 month replayable backups. It enforces a two-phase commit entry ID pattern, creates monthly indices to avoid duplicates, and targets SLOs (99% <100 ms, 75% <30 ms). Monitoring uses Datadog plus a synthetic ping every minute to measure Kafka→OpenSearch latency.
AI-written summary · about
chapters (15)
- 0:00Audit log platform journey
- 1:25Platform requirements and design
- 6:05Kafka vs Elasticsearch for logs
- 7:09Architecture overview
- 10:36Kafka to Elasticsearch flow
- 12:23OpenSearch cluster setup
- 14:50Index lifecycle policies
- 16:38Monthly index routing pipeline
- 18:22Storage and backup strategies
- 19:48Disaster recovery
- 20:31Monitoring with Datadog
- 22:13Security for audit logs
- 23:15Data replay process
- 24:41Production challenges
- 25:01Q&A
related talks
31mMigrating from Self managed Elasticsearch to OpenSearch & How Indias Fantasy Sports Giant Leveled Up
35mApache Kafka as a Monitoring Data Pipeline
38mMigrating from Self-Managed ElasticSearch to OpenSearch
46mOperational Logs and Event Management for HPC S...
37mAuditing your data and answering the question, is it the end of the day yet?
32mBuilding a Real-time Ingestion Platform using Kafka and Aerospike
36mScaling Log Management: PB-Scale Migration From Elasticsearch To...
29mData Acquisition: Overcoming Challenges