Challenges in moving Gen AI apps to production
summary
Company experience moving prototype generative QA into production: built an extractor plus abstractive QA over embedding vectors and layered retrieval (RAG) but finds hallucination unacceptable for safety critical uses like inβcar navigation so deployment stalls. Key blockers are lack of objective evaluation, costly human labeling, and brittle prompt/embedding hyperparameters. A small open source Rilo evaluation tool is used for LLM-as-judge experiments. Building a stable pipeline with multi-tenancy and access controls for cleaned input data proves more valuable than tweaking models.
AI-written summary Β· about
chapters (12)
- 0:00Gen AI production challenges
- 1:08Gen AI adoption in enterprises
- 2:31Query risks in critical systems
- 4:19Balancing risk and accuracy
- 5:25Gen AI in production examples
- 6:31Evaluating Gen AI quality
- 7:52Prompt engineering challenges
- 8:54Human vs AI evaluation
- 10:22Data quality vs model tuning
- 11:47Data pipeline neglect
- 13:15User feedback for Gen AI
- 14:19Closing remarks & Q&A
related talks
41mGenerative AI: bliss or miss?
30mDeveloper Productivity for GenAI Apps
33mOperationalizing GenAI Applications with DevOps Practices
24mScalable GenAI in Production
42mQA for ML: How we can trust AI with Food Sustainability
26mWelcome to the AI Jungle! Now What?
01h 13mBuilding Production-Ready AI Systems in 90 Minutes
55mLessons Learned From Implementing Generative AI Solutions