Data Science from the Trenches
summary
Describes engineering and data‑science work to process ~30 million scientific articles for search, recommendations and expert finding for clients like the European Research Council. Replaced an off‑the‑shelf clusterer with a custom clusterer, packed numeric arrays, released the Python GIL, and streamed DB writes to achieve about 6× improvements in speed and memory. Also covers handling term variants with a specialized stemmer and curated term lists, plus large‑scale author disambiguation to cope with tens of thousands of identical names and ~600k new articles per day.
AI-written summary · about
chapters (6)
- 0:00Scientific publication problem
- 5:19Searching scientific texts
- 14:36Text processing
- 19:53Author identification
- 30:19Author clustering
- 37:03Summary
related talks
38mData science at The New York Times
01h 01mWhat Is Data Science?
01h 00mThe challenges and frontiers of data science in Europe
25mAnusua Trivedi, Data Scientist at Texas Advanced Computing Center (TACC), UT Austin
19mA world of data
19mExploratory Data Science on Federated Raw Data
13mBuilding High-Performance AI Literature Processing
41mData Science – The New Troublemaker in the Delivery Pipeline