Evals First, Code Later: A Practical Guide to Evaluations, Rerankers & Caches
summary
A production code-search engine built with Python, PostgreSQL (pgvector) and OpenAI rerankers reduces average query latency from 20 minutes to 30 seconds — a 40× speed-up — while raising relevance by ≈30%. The workflow starts with task-specific eval datasets and unit tests, uses hybrid (BM25 + dense) retrieval plus cross-encoder reranking, and applies semantic caching to cut embedding time from 24.66 min to 0.61 min. MIT-licensed reference implementation and benchmarked results are available.
AI-written summary · about
chapters (13)
- 0:00Introduction to Evals First
- 4:18Paperpy app architecture overview
- 8:40Synthetic data generation
- 10:12Evaluating search system metrics
- 13:00Answer system evaluation methods
- 14:48LLM-as-a-judge for evaluation
- 17:17Running evaluations in practice
- 18:44Improving with chunking methods
- 20:52Hybrid retrieval and rerankers
- 23:49Semantic caching benefits
- 24:58Query rewriting techniques
- 25:40Fine-tuning with eval data
- 26:03Q&A on evaluator reliability
