The Paradise Papers: Behind the Scenes

on YouTube ↗40m

" summary

13 million documents (1.4 TB) get ingested, OCRed and indexed so 400 journalists across 95 media can search and share leads; processing uses AWS, a distributed extractor called extract, name‑entity pipelines in dataShare feeding Elasticsearch/Solr, and a graph built in neo4j visualized with Linkurious. Security combines GPG, Shemesh single‑sign‑on, 2FA and client certificates. Batch CSV searches, an offshore‑links database published to the public, and impacts include billions recovered by tax authorities and policy changes on company transparency.

AI-written summary · about

" chapters (48)

1. 0:00Introduction
2. 1:01The ICIJ
3. 1:22What is the ICIJ
4. 3:00Public Service Journalism
5. 6:56Its a big leak
6. 7:33Why dedicated
7. 8:25How did we do
8. 8:44Security standards
9. 9:47GPG
10. 10:36HighHub
11. 11:42Security threat
12. 12:49Monitoring
13. 13:55Support
14. 14:26Documents
15. 15:08Emails
16. 15:29Images
17. 15:55Extract
18. 16:32Salar
19. 16:50The Knowledge Center
20. 17:34Batch Search Ease
21. 18:15Open Source Technology
22. 19:05The Data
23. 19:43Scrapers
24. 20:38Craft Database
25. 21:04Offshore Links Database
26. 21:54Influencers
27. 22:17Wilbur Ross
28. 22:40What next
29. 23:36Name Entity Recognition
30. 24:38Data Share
31. 24:57Elastic Search
32. 25:26Datashare
33. 25:56Interface
34. 26:50Microservices
35. 28:21iHub
36. 29:26Funding
37. 30:04How do you find interesting topics
38. 30:57How do you build your tech dev team
39. 31:58Facebooks responsibility to filter fake news
40. 33:09ICIJ membership
41. 33:58Positive results
42. 35:23Security of AWS
43. 36:46Treasureit
44. 37:05Average file size
45. 37:36Trumps face
46. 38:32Attracting good talent
47. 39:13How long did it take
48. 39:49Are the ICIJ under the same amount of protection

" related talks

• 52mUsing Open Source Tech to Unearth the Stories of the Panama PapersMichael Hunger · GOTO Copenhagen 2016
• 14mDatashare + Neo4j: Unstructured Docs to Knowledge Graphs for Investigative JournalismClément Doumouro · NODES 2024
• 33mGovernment Big Data for AllAleksey Ivankin · Highload fwdays'18
• 30mAn architecture for big data: Supporting journalism through transparency in advertisingCatherine Strong, Raman Sarvaria · Big Data & AI Toronto 2020
• 40mArchitectures for Exploring Your Personal DataCandida Haynes · PyData San Francisco 2016
• 19mBig data in the service of reliable newsRadu Pop, Benjamin Dauvissat · Berlin Buzzwords 2023
• 18mLessons from Analysing all of Europe's DataGeorg Wittenburg, Guillaume Aimetti · Big-Data.AI Summit 2022
• 38mIs It A Bug Or Is It A Story?Nicola Hughes · Strange Loop 2016