Time Series and FoundationDB: Millions of Writes/s and 10x Compression in 2000 Lines of Go
Talkat FoundationDB Summit 2019 · on Monday, 2019-11-18· at 10:40· Grand Ballroom 5, Marriott Marquis San Diego Marina·
summary
Prototype demonstrates building a time series engine on FoundationDB that achieves millions of writes/s, 10x compression, and fits in 2,000 lines of Go. It buffers Gorilla-encoded blocks in memory, writes uncompressed commit-log chunks into FoundationDB for durable recovery, and transactionally flushes or merges compressed chunks with per-series metadata and simple zone maps using strict serializability. Recovery reads commit-log chunks and rebuilds in-memory encoders; current prototype requires strictly increasing timestamps and cuts corners for production readiness.
AI-written summary · about
chapters (10)
- 0:00Intro
- 1:04Design Goal - M3DB on Foundation DB
- 1:41M3DB on FoundationDB - Attempt #2
- 4:37M3DB's Architecture - Where's the data?
- 6:13Semi-Stateful Commit Logs
- 7:59Commit Log Cleanup - Code
- 10:41Buffer System - Flush to Foundation DB
- 11:23Buffer System - Transactional Flushes
- 12:42Buffer System - Read Path
- 13:29Conclusion
related talks
22mSolving Everyday Data Problems with FoundationDB
37mTechnical Overview of FoundationDB
34mFoundationDB, the Black Knight
46mFoundationDB: The Best-Kept Secret of New Distributed Architectures!
26mFuture of FoundationDB Storage Engines
50mBuilding the Next-Generation Backend on FoundationDB
30mCatching Waves with Time-Series Data
29mCatching Waves with Time-Series Data