Delighting users with RESTful APIs and DuckDB
summary
Dune turns cached SQL results into queryable local tables by running DuckDB on API servers with Go bindings, translating a controlled API into DuckDB SQL and using an LRU cache for roughly a million per-query tables. Results are stored as Parquet, can load an 8 GB Parquet file quickly, and serve many API queries in sub-millisecond response time. JSON serialization and Go type conversion remain the heaviest cost, with Arrow streaming planned if needed.
AI-written summary · about
related talks
28mAnalyze data easily with duckdb - and the implications on data architectures
01h 02mAnalytics for not-so-big data with DuckDB
38mFrom Big Data to DuckDB: Ryan Boyd (MotherDuck) on Hyper-Tenancy, Duck Lake, and Making Data Fun.
34mData Wrangling with DuckDB
40mDuckDB: Harnessing in-process analytics for data science and beyond
06mDuckDB Applications in Information Retrieval
27mIn Process Analytical Data Management with DuckDB
40mDuckDB beyond the notebook