Lightning Talk: Enabling Hot Restart of Stateful Applications Including GPU-Accelerate...
summary
A Kubernetes operator implements transparent CPU+GPU checkpoint/restore by extending CRIU-based operator workflows and an NVIDIA CUDA utility to dump device memory to host, store images on a PVC, then rehydrate GPUs and CPU state in a two-stage restore. Checkpointing is incremental, asynchronous and compressed to cut transfer time; a demo moves a TensorFlow training job on a T4 during node drain and T4 TensorFlow resumed at epoch 7 with minimal pause. Use cases include spot instance reclaim, node drains, autoscaling and notebook persistence via incremental asynchronous checkpointing.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 0:36Problem Statement
- 2:34How we did it
- 3:54How it works
- 5:26Checkpointing and Restore
- 6:39Enhancements
- 7:42Demonstration
- 7:59Night Chart
- 9:37Other GPU workloads
- 10:07Whats ahead
related talks
37mTransparent, Infra-Level Checkpoint and Restore for Resilient AI/ML Workloads
27mState of Checkpoint/Restore in Kubernetes – Now With GPUs
32mEfficient Transparent Checkpointing of AI/ML Workloads in Kub...
09mLightning Talk: GPU-Scanner: Extending CNCF Observability for Multi-GPU AI Workloads
07mLightning Talk: Slash LLM Cold-Start Times by Pre-distributing GPU...
26mKubernetes and Checkpoint Restore
09m⚡ Lightning Talk: Advanced GPU-Orchestrated Workflows and HPC Integrations on K8s for Distributed AI/ML at Scale
39mKubernetes and Checkpoint/Restore