How to make public data more accessible with "baked" data and DuckDB
summary
Bakes IPEDS into a portable DuckDB file so analysis runs locally or on a tiny web worker, produces about 22 million rows in a 320 MB relational DB, and exposes a Python API plus a Plotly Dash web UI. Pipeline ingests CSVs Excels and HTML dictionaries, handles schema crosswalks, versions each release, and validates outputs. MVP takes a few months; approach trades storage for simpler deployment and fast ad hoc OLAP queries.
AI-written summary · about
chapters (12)
- 0:00Last mile public data problem
- 3:38What is baked data
- 5:05CIPEDs project goals
- 7:12IPEDS dataset challenges
- 9:20Designing CIPEDs for usability
- 11:23Data pipeline behind the scenes
- 13:52Baked data results and benefits
- 15:38Porting data to Colab
- 16:42Interactive web app with DuckDB
- 20:35Benefits limitations use cases
- 22:21Summary
- 22:58Q&A
related talks
28mAnalyze data easily with duckdb - and the implications on data architectures
01h 02mAnalytics for not-so-big data with DuckDB
34mData Wrangling with DuckDB
33mData preparation and analytics on your laptop with DuckDB
25mProcessing Biggish Data with DuckDB and Python
45mAnalytics for not so big data with DuckDB
29m"Serpents and Ducks: wrangling data with Python and DuckDB"
27mIn Process Analytical Data Management with DuckDB