ArcticDB, the OLAP antidote
summary
Claims that OLAP often hurts data science not helps, and that big cloud OLAP is expensive and slower for downloads than smart client‑side reads. ArcticDB is a Python client that stores immutable tiled objects on S3, exposes versioned time travel, and optimizes for time series and sparse wide tables. Demonstrated numbers: a local read of 100 million rows quickly, a backend observed throughput of ~60 GB/s from storage, and a billion‑rows‑per‑second local encoding on SSDs; Bloomberg bquant is already using it.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:27Alternative asset manager role
- 3:12Defining OLAP
- 6:23Server-side vs local analytics
- 8:47Critique of BigQuery
- 13:40Data science filtering needs
- 17:09Low-infrastructure OLAP alternatives
- 19:57ArcticDB design & features
- 23:24ArcticDB consistency without isolation
- 25:32ArcticDB architecture & Python
- 30:06Performance benchmarks
- 31:33OLAP limitations & solutions
- 32:01Q&A
related talks
30mWhat a serverless database means for users
18mOpen-Source OLAP on Open-Source Data Warehouses.
58mUnderstanding Big Data for Software Engineers
28mAnalyze data easily with duckdb - and the implications on data architectures
13mBuild your open data lakehouse with Dremio and AWS (PRT085)
40mDuckDB: Harnessing in-process analytics for data science and beyond
34mData Wrangling with DuckDB
26mColumnar Storage: Redefining Data Management for the Modern Era