Time series collection and processing in the cloud: integrating OpenTSDB with Google Cloud Bigtable
summary
An open source add‑on links OpenTSDB to Google Cloud Bigtable so OpenTSDB uses Bigtable instead of running a full HBase/Hadoop stack; the team builds an OpenTSDB + Cloud Bigtable add-on and an async Bigtable library to emulate OpenTSDB’s async HBase client. Early benchmarks hit ~38,000 events/sec on a single‑node test, initial release 0.2 reached ~12–13k/sec and 0.3 improves performance; integration works but buffering and threading issues remain until a native async client is used.
AI-written summary · about
chapters (14)
- 0:00Intro
- 2:26Storing time series data is a challenge
- 4:29Better alternatives - specialized stores
- 5:52Open TSDB Architecture Server
- 7:37HBase can be too much
- 9:19But all I wanted was a time series database
- 9:35Google Cloud Bigtable
- 11:18Migrate HBase apps to Cloud Bigtable
- 12:49Detour: Asynchronous programming
- 13:59Detour: Why asynchronous?
- 14:36AsyncHBase library
- 15:33AsyncBigtable library
- 16:03AsyncBigtable challenges
- 19:44Future work
related talks
05mNext Big Step
09mExplains the Cloud Bigtable Platform
31mGoogle Cloud Bigtable: petabyte-scale NoSQL database for scalable computation
33mManaging IoT Storage with Google's Cloud Platform
07mTime Series Databases
01h 02mDiamond Sponsor Hall
23mThe Open Source Analytic Universe, Version 2022
39mTimescaleDB: Re-engineering PostgreSQL as a time-series database