Keynote: Into the Black Box: Observability in the Age of LLMs
Talkat KubeCon + CloudNativeCon Europe 2025 · on Wednesday, 2025-04-02· at 09:11· Level 0 | ICC Auditorium·
summary
LLM-backed features are inherently nondeterministic so production success depends on capturing inputs, context and outputs end-to-end: capture prompt and token usage, record how prompt context is constructed, and log any parsing or validation before returning results. Pair concrete eval suites with observability to codify “good” behavior and close feedback loops. Use traces or structured logs to relate user request → prompt generation → LLM span → returned output, and drive SLOs and high-cardinality breakdowns from those traces.
AI-written summary · about
chapters (8)
- 0:00LLM observability challenges
- 2:52Traditional testing fails with LLMs
- 5:18Observability as new testing
- 7:02Evaluations and feedback loops
- 8:28Instrumenting LLM inputs/outputs
- 11:59Debugging LLM responses
- 12:41Modern GenAI observability
- 13:44Optimism in AI chaos
related talks
23mObservability for LLMs: Understanding What’s Happening Under the Hood
17mLLM Observability, Evaluation, Experimentation Platform
29mCustomizing and Evaluating LLMs, an Ops Perspective
35mEvaluating LLM systems: the nuts and bolts
18mBeyond the Chatbot: The Real Impact of LLMs on Observability
32mThe Sorcerer’s Apprentice: The craft of building reliable solutions with LLMs
06mObservability for Large Language Models- 17mLLM Observability, Evaluation, Experimentation Platform