Hugging a Duck: Democratizing Data Access and Exploration with DuckDB and Hugging Face Hub
summary
Hugging Face converts each public dataset to Parquet (up to first 5 GB), adds an index.ddb file containing a DuckDB database and full‑text index, and exposes searches and filters via a REST endpoint that downloads the index and runs a single SQL query. Features implemented: full‑text search, column filtering, and precomputed statistics (histograms, counts) stored in a DB. Pipeline runs at Hub scale (about 100k datasets, hundreds of thousands of daily users ~300k) and shipped in production for ~6 months before switching frameworks.
AI-written summary · about
chapters (9)
- 0:00Hugging a Duck intro
- 1:49Hugging Face Hub mission
- 3:34Dataset access evolution
- 5:42Datasets to Parquet
- 7:28Full-text search with DuckDB
- 9:42Filtering datasets with DuckDB
- 11:50Precomputing stats with DuckDB
- 13:16DuckDB at scale
- 14:42Closing remarks
related talks
27mThe Duck and the DataFrame: A Data Engineer’s Journey with DuckDB
28mAnalyze data easily with duckdb - and the implications on data architectures
01h 02mAnalytics for not-so-big data with DuckDB
25mA quack of all trades: DuckDB, a versatile analytical database to keep in your toolkit
33mData preparation and analytics on your laptop with DuckDB
34mData Wrangling with DuckDB
27mState of the Duck
38mFrom Big Data to DuckDB: Ryan Boyd (MotherDuck) on Hyper-Tenancy, Duck Lake, and Making Data Fun.