⚡ Lightning Talk: Mastering Prefill-Decode-Disaggregated Architecture: Solutions and Best Practices in Alibaba Cloud
Lightningat KubeCon + CloudNativeCon China 2025 · on Tuesday, 2025-06-10· at 16:47· Level 16 | Grand Ballroom I·
summary
Describes a Kubernetes-based solution for disaggregating prefill and decode phases in LLM inference using a LeaderWorkerSet workload, an Ingress Controller, and a node discovery service to deploy Mooncake, RTP-LLM and similar engines without intrusive changes. Covers load-balancing improvements with Envoy and ORCA informed by KVCache and metrics, recommends PD-phase ratios, and addresses production requirements like high availability, elastic scaling, canary releases and observability.
AI-written summary · about
related talks
14mSponsored Session: Lightning Talk: Efficient Inference Serving with Kubernetes Gateway...
39mServing AI at Massive Scale: The Cloud Native Inference Plane Behind Huawei...
08mKeynote: Plug in and Scale: Serving LLM Models on Kubernetes Ma...
42mStop Treating LLMs Like REST APIs
09mLightning Talk: Sailing Kubernetes Operation with AI Power
15mDistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language...
14mAI Deployment: Mastering LLMs with KFServing in Kubernetes
13mLightning Talk: AI Meets Kubernetes: Langflow & GROQ in Automated Incident Management