Monitoring in the era of Generative AI, LLVMs, and embeddings
summary
Argues that monitoring LLMs, vision models, and embeddings scales only when raw data is profiled locally and summarized as compact statistics rather than shipped whole. Demonstrates the open source whylogs logger, shows 4 seconds for 1.2 million rows as a performance point, and describes analytic primitives like data sketches and mergeability for combining summaries. Describes embedding-specific checks (reference anchors, distance distributions), feature‑importance tracing, anomaly detection, privacy via local meta capture, and that the platform runs these pipelines at production scale.
AI-written summary · about
chapters (11)
- 0:00Monitoring modern AI systems
- 1:24Labs and Y-Logs open source
- 3:10AI telemetry and observability
- 9:12Generative AI and foundational models
- 13:24Large language and vision model concepts
- 14:50Embeddings and their space
- 17:19Deploying generative AI models
- 22:16Monitoring training approaches
- 25:03Generative AI boom lessons
- 26:47AI monitoring challenges
- 30:19Q&A
related talks
31mBuilding Trustworthy AI: Open Source Monitoring in Action
10mML-based Log Analysis for Faster Debugging
38mAI Monitoring Mastery: Optimize Performance, Ensure Compliance, and Cut Costs
40mOptimizing AI Applications for Production with Observability
18mAI Observability: How To Fix Issues With Your ML Model
26mML Monitoring & Observability
24mHow OpenTelemetry Helps Generative AI
12mModel Monitoring: Detecting and Analyzing Data Issues