Under 5 minutes to a deployed LLM endpoint
summary
RunPod claims a demo deploy from Hub to serverless LLM endpoint in under five minutes: pick a model, bump the context window, spin up an endpoint on H100s, watch a 41 second cold start as the container initializes and the model downloads, then see ~1.5 second execution for subsequent requests. Pricing is per-worker while handling requests at fractions of a cent per second. Platform metrics: 500,000 developers, 30+ data centers, $120M ARR; hub listings are preconfigured Docker repos.
AI-written summary · about
chapters (7)
- 0:00RunPod introduction
- 2:16Why developers need RunPod
- 4:04RunPod products overview
- 5:30Serverless for real-time inference
- 6:51Deploy LLM via RunPod Hub
- 10:24Interact with serverless endpoint
- 12:10Summary
related talks
33mAccelerating the LLM Lifecycle on the Cloud
44mServerless: Giving superpowers to frontend developers
31mOptimizing Developer Productivity with Serverless Technology
46mServerless for Data Scientists
20mSecure, private LLMs in your cloud with Anyscale Endpoints
50mKa-Ching! How Enterprises Measure LLM Performance and Cost
34mKubeFlow’s Serverless Component: 10x Faster, a 1/10 of the Effort
16mStreamline LLM Fine-tuning on Kubernetes With Kubeflow LLM Trainer