Rapid deduplication and fuzzy matching of large datasets using Splink
summary
Splink is a free Python library that actually produces cluster IDs by combining unsupervised statistical scoring with fuzzy matching and dataset term frequencies, offloading heavy work to backends like Apache Spark or DuckDB so it scales to 100 million+ records and links roughly 1 million records in about a minute on modest hardware. It avoids O(n^2) blowup via blocking rules (50,000 rows would imply ~2.5 billion pairwise comparisons but well-chosen blocks drop that to ~625k), outputs waterfall and match‑weight charts, and ships production‑ready inference.
AI-written summary · about
chapters (15)
- 0:00Deduplication with Splink
- 1:09Why deduplication matters
- 3:37Splink deduplication problems
- 4:18Splink library features
- 6:03Sources for matching
- 10:15Deduplicating real dataset
- 12:00Blocking to reduce workload
- 14:45Configuring similarity comparisons
- 16:10Training model without labels
- 17:11Interpreting match predictions
- 19:39Match weights across columns
- 21:04Generating unique IDs
- 22:05Global Splink use cases
- 23:30Splink resources and next steps
- 24:13Q&A on comparisons and linking
related talks
30mSparkless Patterns: The New Data Stack and Modern Data Architecture
37mSimilarity Search for Product Matching @ Semantics3
39mFaster Analytics for Fast Data with Apache Pinot and Flink SQL
45mAnalytics for not so big data with DuckDB
26mUnleash Big Data Clustering
12mEffortlessly modernize your legacy
31mAnswering Multi-dimensional Queries on Massive Datasets
41mOptimising Apache Spark and SQL for Improved Performance