Beyond Spark MLlib: Deduplicating Common Crawl at Scale
summary
Presents a production pipeline that deduplicates Common Crawl at scale using partition-aware MinHash LSH that drives Spark partitioning, vectorized MinHash with NumPy and pandas_udf to avoid Python UDF overhead, band-based filtering with a MAX_BAND_SIZE to drop boilerplate false positives, and a GraphFrames/local+driver union‑find strategy for transitive closure. Benchmarks deduplicate 253.4M documents, generate ~2.1B candidate pairs and finish under five hours on a 9-node r5d.8xlarge EMR cluster, with a ~30GB peak driver memory for global merge.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 2:55Deduplication importance
- 7:31Key techniques
- 10:28System architecture
- 17:39Benchmark results
- 19:45How to use the open-source library
- 20:26Language architecture
- 23:43Lessons learned
- 33:20Key takeaways
- 33:51Q&A
related talks
25mAccelerating fuzzy document deduplication to improve LLM training w/ RAPIDS & Dask
38mScalable Patient Records De duplication using machine learning
21mCase Law and ML on Spark
24mTaming Data Skew in Production ML Pipelines
35mPySpark : Combining Machine Learning & Big Data
40mData Deduplication using Locality Sensitive Hashing
14mHarnessing Common Crawl for AI and ML Applications
27mData @scale Seattle - Yucheng Low