~/talk

RAGs to Riches: Efficient WebChatbot with Async Scraping & Hybrid Re-Ranking

Talkat PyCon India 2025 · 2025-09-12..09-15· [large-language-models, web-performance]
name:Charan
linkedin:c-s-charan-teja
name:Srikanth
linkedin:srikanthdoddi
slides:none on file

A production RAG pipeline that scrapes a 900+ page site and drops full ingestion from 5h18m to 40m using asyncio 5h18m→40m and incremental SHA‑256 content hashing, extracts PDFs/images/tables with PyPDF2 PyMuPDF and pdfplumber plus Multi-Modal Chunking, embeds into a vector DB, and boosts relevance with Hybrid Re-Ranking (semantic score + page rank + domain weight) plus query fusion/classification to avoid wasted LLM calls; runs weekly with incremental updates.

AI-written summary · about

  1. 0:00Introduction and presenters
  2. 1:08Why we built a chatbot
  3. 3:13What is an LLM and its challenges
  4. 5:20RAG explained with analogy
  5. 8:12Naive RAG pipeline and issues
  6. 10:03Challenges in scraping website data
  7. 12:36Handling dynamic content and variety
  8. 15:55Optimizing content ingestion
  9. 17:24Performance bottlenecks and asyncio
  10. 19:11Improving RAG relevancy and chatbot features
  11. 21:43Hybrid re-ranking mechanism
  12. 23:32Final RAG pipeline walkthrough
  13. 24:17Demo and resources
  14. 25:00Software engineering over AI hype
  15. 26:26Management demo and results
  16. 28:03Q&A on scraping frequency
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.