Building a High-Performance Database with Scala, Akka, and Spark
summary
FiloDB is a production-ready columnar analytics DB built with Scala, Akka Cluster, Spark and Monix that ingests high-volume streams and serves low-latency queries. It achieves billions of ops/sec (a tight loop benchmark hits about 2 billion adds/sec single-threaded) by using off-heap binary records, SIMD-friendly vector operations, JVM inlining, HDR histograms for latency, Akka-based routing/stickiness for real-time ingestion, and Monix/reactive streams for back-pressured IO.
AI-written summary · about
chapters (13)
- 0:00Introduction and speaker intro
- 2:01Why build a streaming database
- 5:35Cassandra’s role and limitations
- 7:23Introducing FalcoDB
- 8:50Why Scala Akka and Spark
- 11:17Reactive stack abstractions
- 15:20Reactive streams and Monix
- 21:04Mixing Akka Cluster with Spark
- 25:47Testing with Akka Multi-JVM
- 26:10Tracing and metrics with Kamon
- 27:56Schema validation with Galactic
- 28:37Machine-speed Scala
- 36:11Summary and Q&A
related talks
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
42mBuilding a Company on Scala
40mProductionizing Big Data - Stories from the Trenches
44mLeveraging Scala And Akka To Build A Distributed Time-series Database
18mScaling Without Limits: KFIN's Journey to Distributed SQL
46mScala and the JVM as a Big Data Platform
47mScala: The Unpredicted Lingua Franca for Data Science
05mInterview