Taming Spiky Log Volumes: Maintaining Real-Time Log Accessibility with Kaldb
summary
Logs spike by 10x increase and cause minutes-to-hours (sometimes over a day) of lag; Slack/airbnb measured freshness SLAs around 50–70% before remediation. The solution uses Kafka + Lucene + S3 on Kubernetes and a backfill recovery indexer that prioritizes fresh data, autoscaling only when catching up. Kaldb advertises petabyte-scale Lucene-based storage, multi-tenancy with per-partition isolation, automatic field-conflict handling, and claims up to 10x cost reduction versus OpenSearch; the system is implemented and running.
AI-written summary · about
chapters (11)
- 0:00Log spikes and real-time visibility
- 2:10What are log spikes and why they matter
- 8:13Causes of log spikes
- 14:12Mitigating spikes at application level
- 18:47Handling spikes in ingestion pipelines
- 22:18Storage strategies for log spikes
- 25:49Introducing Kaldb
- 26:34Kaldb architecture and design
- 30:02Prioritizing fresh logs with Kaldb
- 35:19Kaldb backfill and multi-tenancy
- 38:12Summary
related talks
07mKalDB: Log analytics with DuckDB
40mKalDB: A cloud native log search platform
30mData @Scale Seattle- Sundaram Narayanan
40mLarge-Scale Logging Made Easy
20mHow We Implemented a Near Real-Time Logging Solution Using Fluent Bit And OpenSearch
46mBuilding a scalable logging platform
28mFrom Fragmented Logs to Unified Insights: Scaling O11y
25mOne Size Does Not Fit All: A Polystore Architecture for Logs and Traces