Practical Synthetic Data Strategies for RAG/Agentic Systems
Talkat Open Source Summit Japan + AI_dev: Open Source GenAI & ML Summit + Automotive Linux Summit 2025 · 2025-12-08..12-10·
summary
Synthetic data is used to generate controlled query–answer pairs that preserve distributional relationships from source documents and surface retrieval gaps, with a pipeline that extracts entities, reverse‑generates questions from answers, cleans them, and scores retrievers with BM25 vs dense vectors. Experiments run on news/business corpora and measure retrieval precision and downstream task metrics, enabling faster regression testing and prompt/fine‑tuning cycles. An end‑to‑end loop already produces markdown evaluation reports fed to a coding agent for implementation, though agent prioritization and architecture tradeoffs remain unstable.
AI-written summary · about
chapters (11)
- 0:00Synthetic data for RAG
- 3:56Evaluation for LMM agents
- 7:11Evaluators and verifiers in AI
- 11:50Conditioning synthetic data
- 15:43Synthetic data for search
- 19:24Measuring synthetic data quality
- 21:31Retrieval evaluation pipeline
- 25:41Real-time synthetic data
- 26:45Self-improving AI loop
- 29:18Machine teaching shift
- 30:28Q&A
related talks
28mImproving responses in generative AI applications
25mAdvanced RAG Systems: Scaling Retrieval-Augmented Generation in Production
50mRetrieval Augmented Generation Beyond Toy Demos
30mBuilding a HybridRAG Document Question-Answering System
21mHow RAG is Redefining AI Capabilities
44mRAG for a Medical Company: Technical and Product Challenges AI & Data
11mRAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
27mOptimizing RAG Systems in Practice: Challenges and Proven Solutions