Enabling Seamless AI Workloads: Achieving Zero-Downtime Upgrades for FUSE in Kubernetes
Talkat Open Source Summit Japan + AI_dev: Open Source GenAI & ML Summit + Automotive Linux Summit 2025 · 2025-12-08..12-10·
summary
A production deployment of JuiceFS in Kubernetes achieves zero-downtime FUSE upgrades by preserving the kernel FUSE file descriptor and state across restarts, eliminating the frequent transport endpoint is not connected failures and avoiding GPU work loss. Two concrete methods exist: an in-place binary upgrade that copies a new binary via a Kubernetes Job and signals the service to checkpoint state, and a pod-recreate flow where the CSI node holds the FD and recreates the mount pod. Requires JuiceFS client v1.2+, and offers a dashboard and kubectl plugin for one-click upgrades.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:30File system in user space in Kubernetes
- 5:57FUSE client upgrade challenges
- 13:11Zero-downtime upgrade solution
- 14:58Smooth upgrade mechanism
- 18:08In-place binary upgrade
- 20:19Pod recreate upgrade
- 22:08Upgrade methods compared
- 23:12Triggering upgrades
- 24:37Upgrade best practices
- 25:21Conclusion
related talks
25mAutomating Kubernetes Upgrades for Zero Downtime
26mPains and Gains: Lessons From Orchestrating Volumes for Remote Storage in Cloud Native AI Scenarios
16mZero-downtime upgrades of Flink applications
46mEnd-Users Won't Even Notice, Painless Kubernetes Upgrades
22mCSI Driver X FUSE Drivers: a Kubernetes Object Storage Solution for AI/ML Data Portability
20mAdvanced CSI-FUSE Filesystem for AI/ML Data Management in Kubernetes
29mChanging the Engine Mid-Flight: Zero-Downtime Ceph Upgrades
35mZero-Downtime Deployment on Kubernetes