Mastering AI Issue Debugging: A Data Scientist's Guide
summary
Demonstrates an automated debugging pipeline that ties New Relic metrics, S3-stored JSON payloads and a SQL document store to a single Trace ID lookup, letting engineers see input/output lengths, model card, prompt, retrieved documents and runtime in one view. Prototype uses Azure GPT-3.5 behind a RAG retrieval layer, implemented with LangChain + FastAPI and surfaced via a Streamlit dashboard. Presenter shows a working demo that queries New Relic’s NerdGraph, pulls S3 files named by trace ID and returns correlated records for faster root-cause analysis.
AI-written summary · about
chapters (9)
- 0:00AI issue debugging intro
- 2:27Essential concepts for data scientists
- 4:54Telemetry pipeline vendors
- 9:18Current troubleshooting process
- 10:23Automating debugging workflow
- 13:23Debugging tool demo
- 18:04Dashboard troubleshooting tool
- 26:38Key takeaways
- 27:20Q&A
related talks
40mOptimizing AI Applications for Production with Observability
07mUnderstanding AI performance and cost management with New Relic
16mSewing up success even when RAGs unravel
31mFrom Alerts to Insights: Performing Trace-Based Causation at Scale
40mKeep AI on Track: Guardrails + OpenTelemetry Observability
29mElevating generative AI and LLM observability
48mHands-on guide to monitor your API-driven AI/LLM applications
17mSponsored keynote: Can AI handle the complexity of Kubernetes and cloud-native environments?