Conformance for Inference: How We Reduced Bad Deploys on a GPU Platform
summary
A seven‑step pipeline enforces separate build conformance (CI checks, pinned image SHA and artifact assurance) and serving conformance (GPU preflight, functional cold start, load testing, continuous validation). It catches CUDA/runtime mismatches (example: CUDA 525 vs incompatible driver), validates on Nvidia A100 80GB and MIG profiles using vCluster and NVIDIA AICR recipes, pins images to digests instead of latest, runs synthetic loads (20 concurrent users / 2k tokens example) and prevents OOMs and noisy‑neighbor rollouts before production.
AI-written summary · about
chapters (18)
- 0:00AI inference conformance intro
- 1:45Deployment limits for AI workloads
- 3:50GPU inference stack issues
- 7:22Inference conformance meaning
- 10:58Built vs serving conformance
- 12:20Seven-step conformance pipeline
- 15:07Case study: conformance profile
- 18:57Why simple deployments fail
- 20:43GPU compatibility pre-flight
- 21:46Running vs ready state
- 23:32Load testing and SLOs
- 23:53Conformance contract areas
- 26:39Evidence states and scaling
- 27:42GPU version mismatches
- 28:23Validation with AICR recipes
- 29:05Policy deployment challenges
- 29:47Checklist questions
- 30:09Final takeaway
related talks
38mInference in Progress… Please Monitor Responsibly
27mAdventures in Conformance
12mBuilding Bulletproof AI Platforms
41mPanel: AI Infra Best Practices: Enterprise Do’s and Don’ts
48mBridging the Gap: Writing Portable C++ Programs for CPU and GPU
26mState of the GPU(nion)
33mDeep Dive: CNCF K8s-Conformance WG
41mBeyond Uptime: Revolutionizing Fintech Reliability