Diofanti.org: A Civic Tech Journey Powered by PostgreSQL
summary
Diofanti.org ingests government publication feeds into PostgreSQL, parsing PDFs into JSONB and full‑text columns, indexing millions of records (the government feed counts ~70 million decisions, ~20k/day). Pipeline uses the http extension to fetch data, a custom pgpdf extension plus parsing to store extracted text, and a pgvector-backed semantic search powering a citizen-facing chatbot. It uncovers hidden contracts, drove cancellations, and runs real journalism queries at scale.
AI-written summary · about
chapters (7)
- 0:00Introduction and background
- 3:16What is Diofanti.org
- 7:01Data sources and initial scraping
- 10:13PostgreSQL extensions journey
- 13:18Data workflow and automation
- 16:31Technical learnings and improvements
- 17:35Future features and Q&A
related talks
28mAutomate Your City Data With Python
20mBuild Your Business Based On PostgreSQL Databases
53mModern PostgreSQL for AI...
15mDatasette and Datasette Publish
28mUsing a database in a data science project - Lessons learned in production
40mData Intelligence Platform on AWS
41mAccelerating privacy-enhancing data processing
04mGetting Started with DocumentDB and Open Source