Serving AI at Massive Scale: The Cloud Native Inference Plane Behind Huawei...
Talkat KubeCon + CloudNativeCon + OpenInfra Summit + PyTorch Conference China 2026 · 2026-09-07..09-09·
summary
Describes how Huawei built a cloud-native inference plane for the Celia mobile assistant to serve multimodal AI at massive scale using Prefill‑Decode (PD) disaggregation across tens of thousands of accelerators. Explains concrete Kubernetes-level solutions — PodGroup and SubGroup gang scheduling, custom orchestration of Workload, Router, and AutoScaler — plus routing on tokens and scaling Prefill versus Decode with new metrics, and presents measured traffic/efficiency improvements (e.g., −40% TTFT, +50% throughput and utilization).
AI-written summary · about
related talks
18mKeynote: Optimizing AI Workload Scheduling: Bilibili's Journey To an Efficient Cloud Native AI Platform
18mKeynote: Huawei: Extending the Power of Cloud Native from Cloud to Edge...
05m⚡ Lightning Talk: Mastering Prefill-Decode-Disaggregated Architecture: Solutions and Best Practices in Alibaba Cloud
10mKeynote: Enabling AI with Cloud Native
31mAdvancements in AI/ML Inference Workloads on Kubernetes From WG Serving and Ecosystem Projects
37mEnabling AI For Everyone - Hiai Open Platform Introduction
15mKeynote: From Platforms to AI Factories. Has Kubernetes Solved It?
23mEffortless Scalability: Orchestrating Large Language Model Inference with Kubernetes