HDFS CSI Plugin: Speed Up Kubernetes in On-Premises Big Data Cluster
summary
Shows a CSI-compatible HDFS plugin that mounts on-prem HDFS into Kubernetes using NFS gateway-based mounts and optional Ozone object storage; explains deployment (per-node CSI node daemon plus one active controller service), PV/PVC binding and snapshot/resize limits, and operational tradeoffs for large clusters. Includes a TPC-DS benchmark run to compare Spark on Kubernetes with HDFS versus Spark on YARN with HDFS, plus practical scaling and isolation patterns you can reproduce.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:58Outline
- 3:12Big Data History Cont.
- 3:33Big Data Stack
- 6:19Big Data Trend
- 8:45Benefit of Containerization
- 10:00Kubernetes Architecture
- 10:30Challenges
- 12:12CSI(Container Storage Interface)
- 15:19CSI Core Services
- 16:51CSI Advance Features
- 19:07Volume Lifecycle Volume Lifecycle
- 19:37Controller and Node Services
- 20:51Kubernetes Storages
- 22:17Kubernetes CSI Support
- 25:10PV, PVC and Storage Class
- 27:49Package and Deployment Suggestion
- 29:02Hadoop HDFS
- 30:04HDFS Cluster Scale
- 30:40Apache Ozone
- 31:44HDFS/Ozone as PV
- 34:25HDFS Characteristics as PV
- 36:08HDFS NFS Gateway CSI
- 36:40Ozone CSI
- 37:00Resources
related talks
41mHDFS CSI Plugin: Speed Up Kubernetes in On-Premises Big Data Cluster
45mEmbracing Big Data Workload in Cloud-Native Environment with Data Locality
48mCSI in Kubernetes
24mDynamic Big Data in the Cloud with Kubernetes
35mModern Big Data Pipelines over Kubernetes [I]
32mUnderstanding Kubernetes Storage: Getting in Deep by Writing a CSI Driver
26mBuilding Apache Spark Clusters on K8s for AI Workloads
22mCSI Driver X FUSE Drivers: a Kubernetes Object Storage Solution for AI/ML Data Portability