GammaLake: High-Performance Open-Source Feature Storage for Timeseries
summary
GammaLake is an open-source Point72 project that layers Delta tables with structured tall-and-skinny representations (global sortable index like timestamp+symbol), uses Ray for parallel local orchestration and Polars/Arrow for cheap horizontal concatenation, and runs on Anyscale. Appends avoid full rewrites by per-feature-group parquet/Delta files plus null-padding and Delta time-travel/versioning; reads favor low-latency concatenation over expensive multi-joins. Benchmarks sit in the same ballpark as DuckDB for common workloads, and the implementation is production-tested at Cubist.
AI-written summary · about
chapters (8)
- 0:00Introduction and background
- 10:37Intern's proposal and initial approach
- 14:48Challenges with tall and skinny tables
- 17:58Delta Lake and Arrow format advantages
- 21:52Appending and updating features efficiently
- 23:37GammaLake architecture and Ray integration
- 26:23Recent additions and open-source tools
- 28:51Open-source contributions and closing
related talks
20mDelta Lake: Turn your Data Lake into a Lakehouse
39mDelta Lake: Diving into Data Lakes Without the Downsides
33mThe Pace of Innovation in Delta Lake
18mDelta Lake – The Foundation Of Your Lakehouse
46mOpen Table Formats in the Wild: From Parquet to Delta Lake and Back
19mOpen Source Reliability for Data Lakes
28mLakehouse Architecture with Delta Lake and Databricks
31mDelta Lake: Open Source Reliability and Quality for Data Lakes