~/talk

Evals First, Code Later: A Practical Guide to Evaluations, Rerankers & Caches

Talkat PyCon India 2025 · 2025-09-12..09-15· [search, machine-learning]
slides:none on file

A production code-search engine built with Python, PostgreSQL (pgvector) and OpenAI rerankers reduces average query latency from 20 minutes to 30 seconds — a 40× speed-up — while raising relevance by ≈30%. The workflow starts with task-specific eval datasets and unit tests, uses hybrid (BM25 + dense) retrieval plus cross-encoder reranking, and applies semantic caching to cut embedding time from 24.66 min to 0.61 min. MIT-licensed reference implementation and benchmarked results are available.

AI-written summary · about

  1. 0:00Introduction to Evals First
  2. 4:18Paperpy app architecture overview
  3. 8:40Synthetic data generation
  4. 10:12Evaluating search system metrics
  5. 13:00Answer system evaluation methods
  6. 14:48LLM-as-a-judge for evaluation
  7. 17:17Running evaluations in practice
  8. 18:44Improving with chunking methods
  9. 20:52Hybrid retrieval and rerankers
  10. 23:49Semantic caching benefits
  11. 24:58Query rewriting techniques
  12. 25:40Fine-tuning with eval data
  13. 26:03Q&A on evaluator reliability
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.