Speed Wins: The Basics Of How To Push More Queries Through Each CPU Core
summary
Real case: a 1.1B-row (~30GB) events table returns an 8s query that reads ~42GB; reordering sort keys to event + timestamp and pushing filters first cuts reads to 46M rows (~1.7GB) and latency to ~0.7s. Materializing the join and aggregation with materialized view (Tinybird) yields ~54k rows (~0.002GB) and ~0.003s per query. Customers report similar wins (reads 45GB → 2.5GB, latency 2s → 30ms). Key tactics: filter early, index/sort by query predicates, pre-aggregate incrementally.
AI-written summary · about
chapters (4)
related talks
30mOptimising your Database for Analytics
12mMilliSort and MilliQuery: Large-Scale Data-Intensive Computing in Milliseconds
29mWide Event Analytics
25mData Demons: Unleashing Chaos or Driving Performance?
46mETL Confessions
37mFrom Inverted Index to Columnar Vectorized Execution Search
28mMoving Towards Real-Time OLAP - With First Principle Thinking and Guided by Access Patterns
01h 06mBest practices when accessing Big Data or any other data!