Dealing with TBytes of Data in Realtime
summary
Claim: stitch together hundreds to thousands of single SQL instances into a shared‑nothing cluster and get near‑real production scale. Uses Crate on an unused IaaS cluster and Docker containers, experiments with container‑VM ratios and bulk loading. Reports a working ~1000‑node cluster, users inserting ~3 billion inserts/day and ~0.5M rows/sec search, but with eventual consistency, careful schema design and bulk inserts required for good performance.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 2:17Sensible data and background
- 9:13Demo: distributed cluster
- 18:34CrateDB features and architecture
- 23:07Populating and querying
- 31:30CAP theorem and consistency
- 34:22Performance benchmarks and use cases
- 36:57Q&A: OLTP vs OLAP
related talks
22mIt’s all about the db - a journey to accommodate 100k events per second
36mUnderstanding Databases for Distributed Docker Applications
43mIngesting over 4 million rows a second on a single instance
28m70 TB Servers – How and Why We Use Cassandra
48mBuilding Petabyte-Scale PostgreSQL Deployments
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
13mUnlocking Your AI Potential With Real-Time Data Unification
05mBret Piatt interviewed at Velocity 2010