CSI Driver X FUSE Drivers: a Kubernetes Object Storage Solution for AI/ML Data Portability
Talkat Data on Kubernetes Day North America 2023 · on Monday, 2023-11-06· at 13:30· W471ab, McCormick Place·
summary
A sidecar-based CSI solution mounts object buckets as FUSE filesystems by having the CSI node open /dev/fuse, send the file descriptor over a Unix domain socket, and let a per-pod sidecar receive the FD and start the FUSE driver so the sidecar stays unprivileged and its lifetime and resources match the pod. The demo uses GCS Fuse CSI driver, workload identity, CSI ephemeral inline volumes and trains MNIST in-cluster; limitation: no init‑container support yet.
AI-written summary · about
chapters (17)
- 0:00Object storage CSI FUSE drivers
- 1:02Motivation for the solution
- 2:48Object storage for AI/ML
- 4:15FUSE drivers in Kubernetes
- 5:18Portability and cost benefits
- 7:08Challenges with CSI FUSE
- 9:42Sidecar-based solution
- 11:29GCS FUSE CSI driver architecture
- 12:53Two-step FUSE mount technique
- 14:39Demo setup and Jupyter deployment
- 16:29Exploring mounted bucket data
- 17:12Training job with PVC
- 18:15Training completion verification
- 19:04Inference workload demo
- 19:46Prediction results wrap-up
- 20:10Key takeaways
- 20:52Limitations and future plans
related talks
20mAdvanced CSI-FUSE Filesystem for AI/ML Data Management in Kubernetes
48mCSI in Kubernetes
26mPains and Gains: Lessons From Orchestrating Volumes for Remote Storage in Cloud Native AI Scenarios
32mUnderstanding Kubernetes Storage: Getting in Deep by Writing a CSI Driver
01h 24mTutorial: Introduction to Using the Container Storage Interface (CSI) Primitives
27mCeph-CSI: The Engine Behind Persistent Storage in Kubernetes
41mHDFS CSI Plugin: Speed Up Kubernetes in On-Premises Big Data Cluster
41mHDFS CSI Plugin: Speed Up Kubernetes in On-Premises Big Data Cluster