Data modeling, the secret sauce of building & managing a large scale data warehouse
summary
VeniceDB sits on PostgreSQL + Citus on Azure and handles telemetry from ~1.5 billion devices, serving ~10 million queries/day (≈2 million distinct). Schema uses JSONB for staging, hstore and custom metric hstores for compact storage, and ~50 partial covering indexes per reporting table to trade storage for query I/O. It scales to ~2,000 cores, 15 TB RAM, 1 PB SSD and grows measures from ~100 to ~5,000. Core techniques include device-centric aggregation, approximate counting with HyperLogLog, and percentile merging via T-Digest, delivering multi-second ad hoc queries and months-long uptime.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:19Windows telemetry data challenge
- 4:13Data schema of Venice DB
- 6:28Analytical queries at petabyte scale
- 16:40Venice DB data storage & organization
- 20:30Why Citus?
related talks
55mBreakout Session
24mCitus from the Customer eyes
31mCitus and JSON for real-time analytics at Vizor Games
43mOpen-Sourcing Venice
27mBuilding a PostgreSQL data warehouse
24mScaling to Billions of Internet Observations per Day
46mBig databases, many servers, data center or cloud - it's all here!
25mData-intensive PostgreSQL: Three ways to scale