(Re)training word embeddings for a specific domain
summary
Claims that retraining GloVe / word2vec / fastText on a domain corpus improves domain terms and senses, then shows concrete steps: build a Wikipedia domain scraper, train new vectors or fine-tune pretrained ones (start fixed then unfreeze), run intrinsic checks and an extrinsic classifier benchmark. A small toy experiment (intent classification) finds pretrained and retrained vectors perform nearly the same, and the scraper plus toy app are produced and will be released on GitHub.
AI-written summary · about
chapters (10)
- 0:00Domain-specific word embeddings
- 3:20Word representations basics
- 7:21History of dense representations
- 12:08General embeddings limitations
- 16:48Retraining for domain adaptation
- 20:03Building domain corpus
- 24:24Evaluating embeddings
- 26:39Case study and challenges
- 27:43Key takeaways
- 29:54Q&A
related talks
39mNext generation of word embeddings in Gensim
43mThe Evolution of Domain Transfer in the Era of Pre-trained Language Models
41mFrom Embeddings To Index: A Practitioner's Guide To Domain-Adapted Neural Search
39mSemi-Supervised Bootstrapping of Relationship Extractors
39mBeyond word2vec: GloVe, fastText, StarSpace
31mTaking AI/ML to the next level: Text!
43mDemocratizing NLP content modeling with transfer learning using GPUs
28m"Embeddings: How Computers Learned to Read"