Shipping complex AI applications
summary
Prototype-to-prod gap is the claim and the evidence is concrete: a five-stage agent is built for support triage with local tool calls, run against GPT‑5 Mini and OpenAI keys, traced to token and tool‑call level with Braintrust tracing, and evaluated using a 10‑item golden dataset plus deterministic and LLM‑as‑judge scores. Trainline’s live context (27 millions active users and 6.3 billions tickets) drives online scoring, managed prompts, and a remediation loop that patches prompts, reruns experiments and measures regressions.
AI-written summary · about
chapters (16)
- 0:00Introduction and Welcome
- 4:07Workshop Overview and Agenda
- 4:39Understanding AI Engineering and Operational Challenges
- 9:55Introduction to Braintrust
- 12:56Experience from Trainline
- 28:35Building the Support Triage Agent (Overview)
- 33:57Basic Implementation: Single Shot Prompting
- 40:32Adding Local Tools for Determinism
- 41:30Implementing Specialist Stages (Agentic Flow)
- 46:19Instrumenting and Tracing the Application
- 56:43Evaluating AI Systems and Golden Data Sets
- 1:05:07Deploying and Managing AI in Production
- 1:13:58Online Scoring and Monitoring Production Logs
- 1:19:13Identifying and Remediating Failure Modes
- 1:33:05Key Takeaways and Summary
- 1:36:58Further Resources and Documentation
related talks
40mAI Agents in Practice
34mThe Hard Truth About AI Agents in Production
01h 26mShipping AI That Works: An Evaluation Framework for PMs
32mThe AI Agent Path to Prod: Building for Reliability
43mTesting Agentic AI Applications: Beyond Traditional QA
47mValidation and Observability of AI Agents
30mBuilding Reliable AI Agents: An Open Source Approach To Evaluation and Observability
37mAI agents testing How to evaluate the unpredictable