You Got 10 Million PDFs from the Ministry. Now What?
summary
Processing 10 million ministerial PDF reports into a working public service: the talk covers extracting text from massive PDF collections, detecting and analyzing inconsistencies in the data, and designing a high-performance, low-cost hosting system. It specifically discusses the implementation details and operational challenges behind the slps.pl project, including data cleaning, text extraction pipelines, discrepancy analysis, and efficient service deployment.
AI-written summary · about
chapters (7)
- 0:00Introduction
- 7:09Receiving 10M PDFs
- 11:26PDF processing
- 20:00Building interactive site
- 31:05Data discoveries and issues
- 39:11Unmet system goals
- 42:20Current state and future
related talks
28mAutomate Your City Data With Python
21mStructuring the Unstructured
39mAdventures in Understanding Documents
29mAutomate the creation and manipulation of PDFs using PDF Services API
40mWhat's with the 1s and 0s? Making sense of binary data at scale
43mImplementing AI Document Processing at Scale: Syngenta
13mModernize document workflows with intelligent processing
19mDiofanti.org: A Civic Tech Journey Powered by PostgreSQL