Re-architecting the Data Platform with Spark on Kubernetes
summary
A production data platform migrates logging, Kafka, S3-backed storage, Airflow, Superset and Spark to run on Kubernetes, adds JupyterHub with persistent EBS-backed notebooks and a JDBC/Thrift gateway for SQL clients. The move shortens rollout time (from ~4 months to ~1 month for two engineers), improves self-service and reduces idle cluster operational costs, and supports both batch and interactive queries while noting that dynamic executor scaling and shuffle/executor integrations are still immature.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 3:58Devsisters data platform
- 7:16Spark on Kubernetes
- 10:19Kubernetes benefits
- 13:34Spark standalone issues
- 16:25Spark on K8s deployment
- 18:07JupyterHub + Spark demo
- 21:57Operational improvements
- 23:03Spark on K8s limits
- 24:33Conclusion
related talks
22mSPOK - Managing ML/Big Data Spark Workloads at scale on Kubernetes
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
26mBuilding Apache Spark Clusters on K8s for AI Workloads
17mDemocratising data analytics
40mApache Spark on Kubernetes: A Technical Deep Dive
28mCloud-Native Apache Spark: why & how to migrate your Spark
41mRunning Apache Spark on K8s: From AWS EMR to K8s
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh