Dear Diary: How to Release Official Data from Brazilian Cities with Py...
summary
A project ingests official municipal gazettes from Brazil’s ~5,570 cities, uses OCR to convert PDF notices, Scrapy spiders (about 2,400 written) to extract pages, and indexes the text in Elasticsearch. The system runs spiders twice daily for ~20 production cities, exposes an open API and search site, and uses Telegram monitoring to detect scraper failures; the setup already returns useful search results for journalism and research.
AI-written summary · about
chapters (7)
- 0:00Start
- 6:41Dear Diary
- 9:55Data Extraction
- 13:50Indexing
- 14:28Data Access
- 18:45Documentation
- 24:05Questions?
related talks
24mFreeing Public Data
24mBuilding Open Data Platforms, an Insider's Perspective
52mUsing Open Source Tech to Unearth the Stories of the Panama Papers
35mDesigning for Impact in Civic Data Science
47mPanel: Open Data In Public Use
06mHow Open Data is Making ...
21mThe socio-economic advantages of open data implementations
42mInside Out - Creating a Python Ecosystem for Analysis and Better Government