RAGs to Riches: Efficient WebChatbot with Async Scraping & Hybrid Re-Ranking
summary
A production RAG pipeline that scrapes a 900+ page site and drops full ingestion from 5h18m to 40m using asyncio 5h18m→40m and incremental SHA‑256 content hashing, extracts PDFs/images/tables with PyPDF2 PyMuPDF and pdfplumber plus Multi-Modal Chunking, embeds into a vector DB, and boosts relevance with Hybrid Re-Ranking (semantic score + page rank + domain weight) plus query fusion/classification to avoid wasted LLM calls; runs weekly with incremental updates.
AI-written summary · about
chapters (16)
- 0:00Introduction and presenters
- 1:08Why we built a chatbot
- 3:13What is an LLM and its challenges
- 5:20RAG explained with analogy
- 8:12Naive RAG pipeline and issues
- 10:03Challenges in scraping website data
- 12:36Handling dynamic content and variety
- 15:55Optimizing content ingestion
- 17:24Performance bottlenecks and asyncio
- 19:11Improving RAG relevancy and chatbot features
- 21:43Hybrid re-ranking mechanism
- 23:32Final RAG pipeline walkthrough
- 24:17Demo and resources
- 25:00Software engineering over AI hype
- 26:26Management demo and results
- 28:03Q&A on scraping frequency
