Tales at Scale: Analytics at 1000 QPS and Beyond
summary
Claims Apache Druid delivers interactive analytics at 1000 QPS by combining data layout, cluster design and resource management. Uses columnar segments with dictionaries and block compression, sorting for locality (measured 8x CPU reduction), and deferring expressions to cut per-query CPU. Scales via partitioning, sketches and broadcasting small join tables to avoid data movement; handles heterogenous workloads with priorities, query lanes and optional server tiers. Cites real clusters ingesting 100+ billion rows/day, >1 trillion rows/year, and clusters over 1000 servers with subsecond latencies.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:48Apache Druid
- 3:20Overview
- 4:04CPU IO
- 10:31Locality
- 14:10Deferring computations
- 16:35Scaling
- 21:39Heterogeneous workloads
related talks
38mApache Druid: Why Speed In Analytics Matters
28mThe Heat is On: Architecting for Hot Analytics
42mCasting the spell: Druid in practice
38mCharacterizing the performance of Big Data processing platforms
27mHow to be data-driven when you aren’t Netflix (or even if you are)
53mTips and tricks for robust big data applications
30mData @Scale Seattle- Pawel Terlecki
42mColumnar Storage: Redefining Data Management for the Modern Era