Standing on the Shoulders of Giants
summary
Slack builds Astra, an open source log search engine that ingests 5 million messages a second (≈2 trillion messages over 7 days) to replace 25 Elasticsearch clusters (largest ~1M/sec). It uses a pre-processor → Kafka transaction log → Lucene indexer that writes 30‑minute indexes to S3 and cache nodes that serve queries, while a query node mixes recent indexer data with S3-backed caches. Astra reuses Lucene and OpenSearch for aggregations and APIs, rolls out via Envoy shadow mirroring and a Kibana query‑bridge, and resolves schema conflicts with configurable strategies (drop+mark, coerce to string/int).
AI-written summary · about
chapters (7)
- 0:00Introduction
- 0:42Existing log solution
- 7:12Astra architecture
- 11:08Reusing existing code
- 27:22Incremental improvements
- 28:48Building on shoulders
- 29:31Q&A schema conflicts
related talks
40mTaming Spiky Log Volumes: Maintaining Real-Time Log Accessibility with Kaldb
25mOne Size Does Not Fit All: A Polystore Architecture for Logs and Traces
21mKubernetes Under the Magnifying Glass: Observability With OpenSearch
06mWhy OpenSearch?
41mScaling Observability: From Chaos to Clarity with Open Source Tools
30mData @Scale Seattle- Sundaram Narayanan
24mThe Aryn Journey - Growing a Gen AI Startup on OpenSearch and the Road Ahead
41mMonitoring Solr at Scale