Using Open Source Tech to Unearth the Stories of the Panama Papers
summary
The Panama Papers effort processes 11M documents / 2.6TB with OCR (Nuix on 35 AWS instances), Apache Tika and Solr for extraction and search, then loads a 3.4M‑record dataset into Neo4j + Apache Solr + Apache Tika + Nuix to create relationships and run graph queries (shortest path, PageRank, joint‑investment detection). A lightweight web UI (Lurus/Sigma.js, Linkurious JS, Mapbox) and downloadable CSV/Neo4j DB let non‑developers explore results; the pipeline enables formerly disconnected files to surface publishable stories, built and used successfully by 3 developers supporting 400 journalists.
AI-written summary · about
chapters (25)
- 0:00Introduction
- 1:34What are the Panama Papers
- 2:59Investigative Journalists Consortium
- 6:58Collaboration on leak
- 9:07Data processing workflow
- 10:09Neo4j and Linkurious analysis
- 13:20Power of connected data
- 15:09Why graph databases fit
- 17:15Neo4j basics
- 20:04Modeling and analyzing data
- 22:13Relationships reveal stories
- 24:17Panama Papers data model
- 26:28Visualizing findings
- 27:11Neo4j database interface
- 28:14Running queries
- 30:20Investigative queries and patterns
- 33:31Joint company involvement
- 36:26Geospatial shell company analysis
- 38:15Graph algorithms
- 40:48Connecting data sources
- 41:30Accessible data for non-devs
- 45:04Interactive graph search
- 47:14When to use graph DBs
- 48:40Resources and closing
- 49:02Q&A
related talks
40mThe Paradise Papers: Behind the Scenes
14mDatashare + Neo4j: Unstructured Docs to Knowledge Graphs for Investigative Journalism
40mArchitectures for Exploring Your Personal Data
19mBig data in the service of reliable news
28mSearching for Research Fraud in OpenAlex with Graph Data Science
14mTurning Raw Data into Life-Saving Decisions Using Neo4j and n8n
18mLessons from Analysing all of Europe's Data
33mGovernment Big Data for All