Optimizing LLM Efficiency One Trace at a Time on Kubernetes
Talkat Observability Day North America 2024 · on Tuesday, 2024-11-12· at 12:55· Salt Palace | Level 2 | 255 B·
summary
Deploys a FastAPI service running Hugging Face GPT‑2 on Kubernetes, instruments code with OpenTelemetry eBPF profiler and the Python/Elastic agent, and uses traces to link logs, spans and metrics by Trace ID. Profiling surfaces function‑level hotspots (torch CPU kernels) via flame graphs, shows a 17s text‑generation span and per‑trace carbon share (example 18.74%), and stops blind overprovisioning so GPUs/CPUs and costs drop (authors cite multi‑tens‑percent savings). Demo shows traces, tokens, and profiling working end‑to‑end.
AI-written summary · about
chapters (12)
- 0:00LLM optimization intro
- 1:07LLM deployment challenges
- 4:23Cost and resource management
- 5:05What is OpenTelemetry
- 7:15Profiling for performance
- 9:01OpenTelemetry profiling tools
- 11:31Demo setup with OpenTelemetry
- 15:25Exploring OpenTelemetry dashboard
- 18:43Profiling insights and cost savings
- 21:51Kubernetes optimization metrics
- 22:34Dev to deploy best practices
- 22:56Why observability matters
related talks
27mWho Ate My Resources? Trace LLM Performance in Real Time With OTel
18mHow To Supercharge AI/ML Observability With OpenTelemetry and Fluent Bit
24mWhy Does Continuous Profiling Matter to Developers?
29mKubeflow Trainer Observability: Real-Time Progress Tracking for Reliab...
27mLow-Overhead Tracing Using eBPF for Observability into Kubernetes Apps and Services
01h 00mSystems @Scale Summer Q&A Week 4
19mOops my app performance sucks... what if observability could be a game changer?
32mFrom Traces To Action: Auto-Instrumenting LLMs for Observability...