Combining the Best of Two Worlds: From TF-IDF to Llama LLM
summary
Pipeline ingests years of GitLab support questions, applies spaCy entity ruler and regex rules to extract custom entities like GitLab version (detecting GitLab 15 vs 17), URLs and “cloud native” tech, writes structured JSON, converts texts to embeddings, clusters them, then runs TF-IDF on clustered embeddings to pick the top 3 documents per cluster and pass a narrow context to Llama 2 for human‑readable summaries. System runs in CI/CD, uses small models to cut cost, and drives doc fixes, tutorials and curriculum updates.
AI-written summary · about
chapters (12)
- 0:00Introduction and approach
- 2:35Defining the problem
- 5:47Data analysis methodology
- 7:35Custom entity recognition
- 13:32LLM-assisted annotation
- 15:20Building the NLP pipeline
- 17:35Topic modeling and LLMs
- 23:09Prompt engineering
- 26:07Advanced pipeline extension
- 28:21High-level architecture
- 31:49Outcomes and takeaways
- 34:52Code implementation and wrap
related talks
39mCombining the Best of Two Worlds: From TF-IDF to Llama LLM
29mUsing LangChain and LlamaIndex to customize LLMs...
36mBuilding & Deploying LLM Apps
37mExpanding the Capabilities of LLMs Through Community Collaboration
42mUsing LlamaIndex for building Generative AI apps
40mInstructLab: Applying Open Source Methods to Building and Trai...
49mBen Hall & Barbara Fusinska- 37mExpanding the Capabilities of LLMs Through Community Collaboration