One Size Does Not Fit All: A Polystore Architecture for Logs and Traces
summary
Claims a polystore routing logs and traces to specialized engines (Lucene/OpenSearch, DuckDB) via Kafka and S3, with a unified OpenSearch-compatible API and automated engine selection. Architecture uses dual indexing and a multi-threaded recovery task to prioritize fresh data and backfill spikes. Runs at petabyte scale in production at Airbnb and Slack, reports show up to 30–70% cost savings and pilots surface up to 100× more errors by ingesting more data, proving the approach works.
AI-written summary · about
chapters (17)
- 0:00Observability challenges
- 1:04Human vs agentic workloads
- 3:29Single-engine observability fails
- 7:22DuckDB vs Elasticsearch performance
- 9:52CalyDB polystore architecture
- 10:37CalyDB storage design
- 12:26Indexing vs serving separation
- 13:31Multi-tenancy in CalyDB
- 15:15Handling log spikes
- 16:59Dual indexing benefits
- 17:41Schema-less conflict resolution
- 18:44Long-term retention with S3
- 19:06OpenSearch API benefits
- 20:09CalyDB production readiness
- 21:35Dual indexing demo
- 22:59Automated performance reports
- 24:01Conclusion and future
related talks
40mTaming Spiky Log Volumes: Maintaining Real-Time Log Accessibility with Kaldb
21mKubernetes Under the Magnifying Glass: Observability With OpenSearch
43mBeyond Logs: Real-Time Anomaly Detection with Machine Learning
45mTracing: A journey to tactical insights
19mFirst Principles of Observability Systems
06mWhy OpenSearch?
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
40mObservability that scales: Analyze data in stream and query remotely