Storage Requirements and Options for Running Spark on Kubernetes
summary
Concrete choices decide whether Spark on Kubernetes works: use mtdir for local scratch (host SSD tied to pod lifecycle) for high IOPS, use PVCs/CSI or NFS for shared mounts, and prefer object storage via S3 connector not POSIX mount for final persistence with Alluxio as an optional cache. Use init containers or shared container FS for large user libraries (can reach 8–10 GB), push logs via a sidecar (never write logs to container FS), and run Kubernetes as the resource manager to eliminate extra Spark daemons and enable fast on‑demand runs.
AI-written summary · about
chapters (13)
- 0:00Introduction
- 1:15Speaker intro
- 4:58Spark overview
- 7:03Run Spark on K8s
- 13:14K8s benefits for Spark
- 15:48Spark on K8s deploy options
- 19:31Storage requirements for Spark on Kubernetes
- 25:35Key storage traits
- 30:11K8s storage options
- 32:38Storage solutions for each Spark requirement
- 42:20Persistent storage tips
- 43:23Summary
- 44:06Q&A
related talks
39mSpark on Kubernetes: Best Practice and Performance
29mDynamic Large Scale Spark on Kubernetes
28mCloud-Native Apache Spark: why & how to migrate your Spark
21mSpark on Kubernetes: The Elastic Story
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes
40mApache Spark on Kubernetes: A Technical Deep Dive
26mBuilding Apache Spark Clusters on K8s for AI Workloads