Large Scale, Solr & HBase Performance Monitoring
summary
A hosted SPM monitors Elasticsearch, Solr and HBase by shipping a collectd agent plus JVM instrumentation to a Jetty receiver, feeding a Flume → HBase pipeline that writes a raw table and pre-aggregates into an aggregate table for fast queries. Alerts run on an Esper-based CEP collector, aggregation uses MapReduce when needed, and the deployment handles over 8.9 billion raw rows and ~10k+ datapoints/sec. Operations tuning (keepalive, swappiness, ulimits, async HBase client, HBase-hut pre-aggregation) proves critical and moving to a larger EC2 instance yields a surprising ~20× speedup.
AI-written summary · about
chapters (9)
- 0:00Introduction to SPM
- 2:11SPM architecture overview
- 6:32Agent and collector components
- 9:17HBase and alerting setup
- 11:30Frontend and visualization
- 13:36Live demo and metrics example
- 14:39Lessons from scaling SPM
- 17:50HBase optimization tips
- 18:52Survey results and closing
related talks
53mBig Data: from mammoth to elephant
55mApache HBase: crazy dances on the elephant back
36mSources, Sinks, and Operators: A Performance Deep Dive
29mRealtime Analytics Unleashed: Unlocking the Potential of Data at Scale
30mData @Scale Seattle- Sundaram Narayanan
38mStreaming analytics and CEP ... #bbuzz
36mReal Time Big Data Analytics with Kafka, Storm, HBase #bbuzz
41mBig data as we know it @bol.com