Record linkage: Join for real life
summary
Record linkage is a probabilistic naive Bayes match that actually builds probabilities, uses hidden Markov models to parse names/addresses, and applies blocking (standard, sorted‑neighborhood, q‑gram) to avoid N‑squared compares on 50 million records. Practical tooling includes freebill which outputs editable scripts and runs parallel comparisons that report about a 10× speedup on 15 processors. Real outcomes: deduplication and geocoding reduce missing addresses (roughly 20% in one example) but about 1% unresolved remain and require clerical review.
AI-written summary · about
chapters (22)
- 0:00Start
- 1:20Welcome to the data linkage
- 4:16Freely extensible biomedical record linkage (febrl)
- 5:03Do two records refer to the same person
- 5:21How many people?
- 5:56The steps
- 6:49Data standardisation
- 7:39Hidden Markov Models (HMM)
- 9:47Types of blocking
- 10:02Standard blocking
- 10:34Sorted neighbourhood
- 11:50Q-grams
- 12:51Multiple blocking runs
- 14:04Closeness of dates
- 15:10Closeness of addresses
- 15:41Closeness of names
- 17:03Similarity score
- 18:32Master linkage key
- 20:18Same person? Maybe?
- 21:24Ethics and law
- 23:18Example of geocoding
- 26:09References
related talks
45mCustomizable probabilistic record linkage with Name Match
29m1 + 1 = 1 or Record Deduplication with Python
41mSimilar entity detection in large data
29mBig Data Deduplication and data matching using Python
35mBreakout Session
40mBridging voter data and social data using Apache Streams
53mBreakout Session
24mMachine learning techniques for data cleaning