Mastering Prompts with Feedback and Pydantic
summary
A productionized prompt optimization pipeline uses LangGraph for orchestration, Olama for local model runtime, Streamlit for UI and Pydantic validation as a hard guardrail to enforce JSON schemas and reject malformed fields, with an LLM-as-a-judge multi‑metric evaluator driving an automated feedback loop. A support‑ticket extractor moves from brittle manual tweaks to iterative refinement, the headliner demo reaches target quality on the second iteration, and the system saves validated prompts once thresholds are met.
AI-written summary · about
chapters (22)
- 0:00Introduction and speakers
- 1:11Why prompting matters today
- 3:45Prompting techniques overview
- 5:53From trial and error to engineering
- 7:18Building a feedback loop
- 8:45Evaluation framework with LLM-as-judge
- 9:51Introducing Pydantic for validation
- 11:57Pydantic validators in detail
- 13:29Pydantic as a guardrail
- 14:32System implementation overview
- 15:38Architecture step by step
- 17:04Live demo walkthrough
- 18:32Key learnings and takeaways
- 19:41Q&A session begins
- 20:12Handling Pydantic validation failures
- 21:35Avoiding overfitting in evaluation
- 23:00DSPy comparison discussion
- 24:23Setting evaluation thresholds
- 26:33Choosing LLM-as-judge models
- 27:40Pydantic vs LLM-as-judge roles
- 28:46Cost considerations with LLM-as-judge
- 29:10Ensuring reproducible LLM responses
