Leveraging SRE and Observability Techniques for the Wild World of Building on LLMs
summary
Honeycomb builds a query assistant on top of OpenAI APIs with about six weeks development and eight weeks iteration, instruments every prompt and response, and uses traces and telemetry to compare inputs to outputs. Teams set an initial 70% SLO and track qualitative success signals (accept+iterate, latency, user flows) to prioritize fixes. Measuring from the outside in exposes hallucinations, surfaces which context fields hurt or help, and the feature meets its retention and usability goals in production.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:25LLM integration magic
- 4:35LLM system unpredictability
- 8:11Observability for LLM reliability
- 9:15Honeycomb LLM feature case study
- 12:28Real-world LLM lessons
- 13:10Instrumentation as docs and tests
- 16:44Debugging LLM traces
- 17:45Developer shift to production
- 20:16SLOs for LLM improvements
- 23:05Customer LLM observability
- 25:11LLM development techniques
- 26:19Q&A
related talks
50mObservability and the Development Process
15mKeynote: Into the Black Box: Observability in the Age of LLMs
16mTaming Rogue AI Agents with Observability-Driven Evaluation
45mObservability is for User Happiness
46mEnhancing Agent Reliability Through Evaluation-Driven Development
21mAI-Powered O11y with LLMs: Resilience in Cloud PaaS Archs
17mLLM Observability, Evaluation, Experimentation Platform
32mEvaluating multi-turn conversations