Making sense of Cape Town using NLP
summary
City HR job descriptions become numeric: ~100,000 rows for ~15,000 positions are embedded with the spaCy large model into 300-dim vectors, reduced per person by a center-of-mass weighting of criteria groups, and scored against query terms using cosine similarity. Analysis takes about two minutes on 16 cores to vectorize and ~30–40 hours of work overall; PCA of plotted positions explains ~30% in two dimensions, revealing three bands of data intensity and gaps in gathering/dissemination.
AI-written summary · about
chapters (16)
- 0:00Introduction and context
- 1:51City data strategy and HR problem
- 5:26Data cleaning and Spacy setup
- 10:23Vector space embedding process
- 15:00Visualizing job structure in space
- 16:27Measuring data-related work intensity
- 21:23Validation and key findings
- 23:51Recommendations and next steps
- 24:59Q&A: Analysis time and resources
- 26:26Q&A: Data collection challenges
- 27:30Q&A: Pre-trained models and validation
- 29:54Q&A: Multi-intent and entity detection
- 31:19Q&A: Manifold projections and insights
- 34:12Q&A: Word selection and job gaps
- 37:21Q&A: Job trends over time
- 38:46Closing and acknowledgments
related talks
33mUsing the Amazing World
01h 01mWorkshop: Intro to topic modeling using Python
01h 38mNavigating the World of NLP and Text Analytics
33mScalably Internationalizing Millions of Latent Semantic Labels
37mBuilding a Scientific Taxonomy at Scale
40mDesigning spaCy: Industrial-strength NLP
31mTaking AI/ML to the next level: Text!
59mContextual search with vector search: exploring your options with open source tools