Accelerating fuzzy document deduplication to improve LLM training w/ RAPIDS & Dask

on YouTube ↗25m

" summary

NVIDIA NeMo Data Curator uses RAPIDS and Dask to scale fuzzy deduplication of terabytes of text by bucketing with MinHash, expanding pairs into a Dask dataframe, and computing Jaccard similarity on GPU‑accelerated partitions. The workflow runs on a RAPIDS + Dask cluster deployed on Kubernetes and reports dramatic speedups for a 4.5 TB dataset, for example 4 nodes with GPUs finish in 3 hours vs 20 CPU nodes in 37 hours, showing RAPIDS + Dask on Kubernetes cuts time and cost.

AI-written summary · about

" chapters (8)

1. 0:00Introduction
2. 2:48LLM data preprocessing challenges
3. 5:13RAPIDS GPU acceleration tools
4. 7:23Scaling with Dask
5. 11:18Cloud deployment
6. 15:30Fuzzy document deduplication workflow
7. 21:06Performance gains
8. 22:49Q&A

" related talks

• 01h 22mHigh-Performance Data Science at Scale with RAPIDS, Dask, and GPUsKeith Kraus · PyData New York City 2019
• 38mRAPIDS- Open Source GPU Data ScienceBrad Rees · PyData Los Angeles 2019
• 39mAccelerating Data Science with RAPIDSKeith Kraus · PyData DC 2018
• 01h 08mGPU Data Science with RapidsKeith Kraus, Joshua Patterson · PyData Miami 2019
• 41mAccelerating Data Science with RAPIDSMike Wendt · PyData Los Angeles 2018
• 34mRAPIDS: How to accelerate your data science pipeline by orders of magnitudeAdam Grzywaczewski · PyData Eindhoven 2020
• 01h 36mRAPIDS: GPU-Accelerated Data Science for PyData UsersNaty Clementi, Mike McCarty · PyData Virginia 2025
• 20mEnd-to-End Data Science on GPUsJohn Zedlewski · OpML '20