The Hater's Guide to Dealing with Generative AI

on YouTube ↗29m

" summary

Claim: production generative systems succeed or fail on observability not model magic. Built a natural-language→Honeycomb query feature using gpt-3.5 in production and a RAG pipeline with vector embeddings and chunking. Instrumentation logs the full prompt, response and errors, stitches upstream retrieval into distributed traces (26 high-level ops), and tracks SLOs. Iteration raises valid-query rate from about 70→96% accuracy. Practical steps: prototype in ChatGPT, put prompts under source control, log full prompt, and use OpenTelemetry-style traces to debug retrieval and postprocessing.

AI-written summary · about

" chapters (15)

1. 0:00Introduction
2. 0:38AI Hype
3. 1:25Crypto vs AI
4. 2:51Augmentation vs Automation
5. 4:12AI Reliability Overlap
6. 5:36AI Applications
7. 11:28Data and Search
8. 13:05Making a Bad AI App
9. 13:51Making a Better AI App
10. 15:24Log the Response
11. 15:57Turn Logs into Traces
12. 19:03Hot Tips
13. 21:36ML Engineers
14. 22:05Production
15. 25:00QA

" related talks

• 15mHarnessing the Power of Generative AI for Intelligent ApplicationsShivali Naik · Conf42 Machine Learning 2025
• 38mExploring ChatGPT for Improved ObservabilityGareth Emslie · NDC Porto 2023
• 41mThe Unreliable Computer Revolution - Embracing the Double-Edged Sword of GenAINoz Urbina · Web Summer Camp 2024
• 24mHow OpenTelemetry Helps Generative AIPhillip Carter · OTel Community Day North America 2024
• 37mGenerative AI: Beyond (Just) HypeTodd Underwood · SREcon24 Europe/Middle East/Africa
• 44mExploring ChatGPT for Improved ObservabilityGareth Emslie · DevOps Barcelona Conference 2023
• 41mConnect GPT with your data: Retrieval-augmented GenerationMalte Pietsch · Berlin Buzzwords 2023
• 45mHow to use GPT in your projects: A Case StudyStephan Janssen · Devoxx France 2023