Cleaning Messy Data at Scale: APIs, LLMs, and Custom NLP Pipelines
summary
Example uses 5 million raw US addresses matched to a 120 million ground truth; simple left join starts at 47% match rate, adding regex + open-source parser (postal) raises it to 76%, buying a 3rd‑party address API jumps to 90% quickly but with vendor lock and cost, large-cloud embeddings (Gemini via BigQuery) reach ~89%, fine‑tuned MiniLM L6 gives ~83%, and a custom 5M‑parameter model trained on 40M pairs with contrastive + MNR loss hits 88%; a waterfall pipeline (join → parsing → embeddings → API) balances cost and accuracy.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 1:04Why address matching
- 5:16Third-party API solution
- 10:34Rule-based parsing
- 19:52Semantic embeddings
- 29:47Fine-tuning models
- 36:56Custom NLP pipeline
- 45:20Comparing solutions
- 47:51Q&A
related talks
24mMachine learning techniques for data cleaning
33mDeep research at scale
28mThe Data That Shapes Foundational LLMs
17mFrom messy to meaningful data: LLM-powered classification in R
51mQuery Understanding With Large Language Models: Techniques and...
01h 19mBuilding an LLM on your Laptop to...
40mTechniques to avoid common LLM pitfalls
33mLessons Learned Applying Large Language Models in Healthcare