FiloDB: Real-time, In-Memory Time Series at Massive SMACK Scale
summary
FiloDB is an open-source, in-memory time-series database built on Scala, Akka, Kafka, Cassandra and Mesos that targets high-cardinality monitoring and container use cases. It stores columnar, delta-encoded blocks off-heap with custom zero-allocation ingest structures, persists raw input to Kafka and compressed columns to Cassandra, and serves Prometheus-style queries via a distributed Akka-based executor. A single node demonstrates ~1M points/sec ingest and ~3 bytes/sample compressed footprint while queries run directly on compressed columnar data without full decompression.
AI-written summary · about
chapters (18)
- 0:00Introduction to time series
- 7:30Introducing FiloDB
- 11:25Time series problem and solutions
- 13:32Dataflow architecture
- 15:01Ingestion and backpressure
- 16:28Columnar compression techniques
- 19:20Performance benchmarks
- 20:24Heap and off-heap optimizations
- 24:42Query layer and PromQL
- 27:27Physical query execution
- 29:56Dynamic query optimization
- 31:04Data model and cardinality
- 33:35Multi-tenancy and datasets
- 34:38Persistence and recovery
- 35:00Clustering and shard management
- 36:51JVM pros and cons
- 38:18Tooling and open source
- 39:40Q&A session
related talks
40mThe Seven Deadly Sins of Streaming
16mScaleDB: A Scalable, Asynchronous In-Memory Database
41mIntegrating Kafka with a Real-Time Database
47mBuilding a Real-Time Analytics Database
49mIntroduction to Real-Time Analytics with Apache Pinot
05mClose Encounters with the Third Kind of Database
39mFaster Analytics for Fast Data with Apache Pinot and Flink SQL
42mContemporary Approaches to Data at Scale