A case study on building our first LLM feature – how to balance speed + quality
summary
A case study of building a production LLM feature in 1 month by a two‑person team: a pipeline that scores code‑review feedback for tone and specificity, trained on ~1,000 engineer‑labeled examples (fewer than ten toxic negatives), evaluated to prioritize precision on negative labels and P99 cost per task, uses BAML, Gradio, Streamlit, Weights & Biases and mixes Mistral and Claude models — outcome: persistent top‑five feature with ‹1% extreme misclassification and zero hallucinations.
AI-written summary · about
chapters (15)
- 0:00LLM feature case study intro
- 1:04Feedback quality in code reviews
- 7:03Learning from experts quickly
- 9:30Symbol tuning for classification
- 10:32Industry feedback and tool selection
- 11:55Modeling approach and evaluation
- 15:46Good vs bad products via evaluation
- 17:09Ground truth with domain experts
- 18:55Evaluation metrics for UX
- 20:42Reducing experimental overhead
- 21:46Model biases and limitations
- 22:51P99 cost per task budgeting
- 23:12AI ethics in data labeling
- 23:55Guardrails and transparency
- 25:16Key takeaways and user feedback
related talks
01h 49mAI engineering with open access LLMs that lie, curse, and steal
25mEngineering Better Evals: Scalable LLM Evaluation Pipelines That Work
43mAI Development Lifecycle: Learnings of What Changed with LLMs
33mDesigning LLM Applications for Massive Datasets
38mMaking LLMs reliable - A practical framework
29mAutomating Privacy Reviews with LLMs: Challenges & Opportunities
38mNobody Pages the LLM: Engineering Rigour for Vibe Coding
56mDehumanizing Agents: Why Explainability is Crucial in the LLM Era