Tales at Scale: Analytics at 1000 QPS and Beyond
summary
Explains how Apache Druid achieves thousands QPS with concrete techniques: columnar segmented storage with dictionaries and indexes that avoid reading unused columns, locality via sorting that yields measured 8x CPU and I/O reduction, partitioning to limit servers touched, and cluster techniques like sketches and pre-broadcasted join tables to cut data movement. Describes resource controls—query lanes and server tiers—and operational tools (query profiling, deferring expressions, speculative requests) plus real cluster scale: 100+ billion rows/day, trillion-row retention, clusters over 1,000 servers.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:48Apache Druid
- 3:20Overview
- 4:04CPU IO
- 10:31Locality
- 14:10Deferring computations
- 16:35Scaling
- 21:39Heterogeneous workloads
related talks
32mSome Like It Hot
27mSucceeding with Apache Druid and Clickstream Data
39mApache Druid 101
28mBest of Stream + Batch: When Streaming Analytics Isn’t
28mThe Heat is On: Architecting for Hot Analytics
27mCasting the Spell: Druid in Practice
45mHow We Got Druid to Work at Odnoklassniki
20mAnalyzing streaming data with Apache Druid