Distributed COUNT(DISTINCT) with HyperLogLog on PostgreSQL

on YouTube ↗49m

" summary

Shows how to implement approximate distinct counts in PostgreSQL using the HyperLogLog algorithm, and how to make those counts work across a distributed setup. Covers using HyperLogLog data structures (likely via extensions), merging HLL state across nodes, and integrating that into PostgreSQL queries to provide scalable COUNT(DISTINCT) estimates.

AI-written summary · about

" chapters (4)

1. 0:00Start
2. 15:59Error rate of HLL is damn good
3. 23:32How does HLL work? Stochastic Averaging
4. 33:50How to merge COUNT(DISTINCT) with HLL

" related talks

• 12mScalable Uniques in Postgres - HyperLogLogCraig Kerstiens · PostgresOpen 2013
• 26mBreaking Down Our Billion User Reach with HyperLogLogDaniel Avramson · Reversim Summit 2019
• 28mBreakout SessionGavin McQuillan · PGConf Silicon Valley 2015
• 35mBreakout SessionAJ Welch · PGConf Silicon Valley 2015
• 52mA Deep Dive into Postgres StatisticsLouise Grandjonc · PostgreSQL Conference Europe 2024
• 45mSELECT 'amazing_features' FROM "postgresql"Kevin Davin · Voxxed Days Ticino 2024
• 45mOK Log: Distributed and Coördination-Free LoggingPeter Bourgon · SREcon17 Europe/Middle East/Africa
• 38mAdd ALL the things: abstract algebra meets analyticsAvi Bryant · Strange Loop 2013