Monitoring Solr at Scale
summary
Bloomberg’s Search Infrastructure describes a monitoring stack that scales from hundreds to thousands of Solr services by separating collection of state from alerting: Kafka decouples monitoring and alarming, rules run separately and tenants can consume events. A Night Owl service continuously polls live nodes, clusterstate and leader status; Watchtower and a UI expose per-tenant views. A Slack ChatOps bot surfaces status. Concrete wins include detecting nightly recovery spikes caused by CMS GC and resolving them by switching to G1, and running monitoring components in Kubernetes for horizontal scale.
AI-written summary · about
chapters (16)
- 0:00Introduction and context
- 1:13Why monitoring matters
- 3:00Bloomberg and Solr scale
- 5:25Solr cloud terms explained
- 7:15Monitoring vs metrics
- 11:55Zookeeper monitoring
- 14:50Visualizing Solr data
- 17:24Night Owl monitoring
- 18:37Watchtower dashboard
- 23:36Slack chat bot BFSbot
- 25:43Shard loss scenarios
- 27:48Leadership loss handling
- 28:54Alarm infrastructure
- 31:19Kubernetes monitoring
- 32:24Future plans and open source
- 33:26Q&A session
related talks
12mA Principled Approach to Monitoring Streaming Data Infrastructure at Scale
51mMonitoring and Alerting for Apache Solr with the Prometheus Stack
45mDecentralizing SaaS Monitoring Using Open-Source Tools and Standards
21mChallenges, Best Practices, and Solutions for Monitoring and Alerting...
46mOperational Logs and Event Management for HPC S...
14mKey Considerations For Success When Deploying A Solution For Best-of-breed Observability At Scale
31mServerless Observability: a case study of SLOs
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution