The necessity of pipelines in Natural Language Processing
summary
Argues that reproducible pipelines are the only practical way to manage dozens of preprocessing choices, demonstrating code that wraps tokenizers, cleaners, stemmers and vectorizers into a Scikit-style pipeline and serializes variants to disk. Uses SkLearn and NLTK alongside mentions of Spacy and GenSim. Shows a simple combinatorics example (about 36 pipeline variants, earlier experiments produced ~65 floating variables) and reports that persisting trained pipelines enables hands-off grid-search across preprocessing and modeling and makes workflow usable.
AI-written summary · about
chapters (10)
- 0:00NLP pipelines introduction
- 1:43Why pipelines feel broken
- 3:28Feature extraction in NLP
- 8:20Preprocessing choices explosion
- 10:49Rigid pipelines fail on data types
- 13:16Flexible NLP pipeline design
- 17:30Simplifying pipeline management
- 20:16Pipelines with ML models
- 22:23NLP workflow takeaways
- 24:12Q&A
related talks
38mBuilding Data Pipelines in Python
33mUsing Scikit-Learn's interface for turning Spaghetti Data Science into Maintainable Software
01h 28mMaintainable code in data science
40mBuilding new NLP solutions with spaCy and Prodigy
28mGoing From Notebooks to Scalable Systems
18mReality is not an end-to-end prediction problem
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
56mML Pipeline: Structuring Data Analysis Projects