Lessons from Analysing all of Europe's Data
summary
A system autonomously crawls EU and national portals, processes 140,000 CSVs in eight days on 256 Azure instances and averages 4 minutes per dataset, using a hybrid expert architecture with 428 classifiers plus neural nets. The pipeline extracts time series, geo and statistical signals, flags surprises (one deployment found previously unknown banking fraud) and reaches the quality of a junior data analyst, scaling to tens of thousands of datasets per day.
AI-written summary · about
chapters (4)
- 0:00Intro
- 0:40Lessons from analyzing all of Europes data
- 7:31Questions about data analysis
- 11:06What makes a good analysis
related talks
30mProcess large-scale data and computational function in serverless manner
01h 02mData lakes beyond clusters
37mOPEN DATA - THE NEW GOLD WITHOUT THE RUSH
34m.NET: Analysis and Processing of Large Datasets (...)
29mAI-Powered Anomaly Detection in Cloud Data Pipelines
01h 36mSpark for Scala Developers
41mDaniele Montesi & Emanuele Viglianisi
55mSocial Listening for Richer Data, Unstructured Analytics