A Tale of One Billion Time Series
summary
Baidu stores and serves 1 billion time series with a three-tier TSDB: Redis+HBase+HDFS for hot, warm and cold data; ingestion hits ~40M points/sec with 50 TB/day and read/write throughput ~10M reqs/sec. The system adds multi-level down-sampling written into HBase, Redis caching of hot/key metrics, batch async writes and partitioned HBase tables to avoid compaction/split stalls. Response 99p falls ~90% and cluster-level hot-standby and repair jobs preserve availability.
AI-written summary · about
chapters (30)
- 0:00Intro
- 0:22About me
- 2:00Want a chicken dinner?
- 4:22Everything about our services
- 5:08Incident response
- 5:32Capacity planning
- 7:24What 1B time series will bring...
- 11:12Data layout in HBase
- 12:30Layout of data table
- 14:20Problems: slow queries
- 15:46Query patterns
- 16:38Query cases
- 17:41For latency-insensitive queries
- 19:00For latency-sensitive queries
- 20:14For long-term trend data
- 21:34Multi-level down-sampling
- 24:06Distinguish key & hot metrics
- 24:37Cache key & hot metrics in Redis
- 25:11Reduce Redis memory consumption
- 25:50Problems HBase performance degradation
- 26:22About compaction
- 28:11Why are compaction & splitting bad?
- 30:55Region pre-splitting
- 32:49Reduce HDFS R/W latency
- 34:03Deal with stop-the-world GCs in saver
- 36:37Final architecture
- 37:34Hot standby
- 38:47Multi-level data availability
- 39:34Isolation
- 40:55Summary
related talks
30mCatching Waves with Time-Series Data
22mIt’s all about the db - a journey to accommodate 100k events per second
07mTime Series Databases
51mPostgreSQL, The Time-Series Database You Want
29mCatching Waves with Time-Series Data
30mScaling Time-Series Data to Infinity: A Kubernetes-Powered Solution with Envoy
49mIoTDB - Moving from Relational to Timeseries
29mDistributed SQL: How TiDB Modernized Data Management