Spark Operator - Feature Engineering with Spark on Kubeflow
summary
Shows a reproducible setup that runs Apache Spark via the Spark Operator on Kubeflow Pipelines, submits spark-application YAMLs with kubectl, and returns driver logs to MinIO/S3 so non‑K8s users can see outcomes. Demonstrates a Spark Pi end‑to‑end demo that succeeds, and real work parsing PDFs with Apache Tika and Tesseract at scale — roughly 1.2 TB of historical PDFs and ~800–900 GB per year processed. Notes concrete limits: job arguments aren’t visible in the UI and STO sidecars can block pipeline progress.
AI-written summary · about
chapters (7)
- 0:00Spark Operator on Kubeflow
- 3:14Why Spark on Kubernetes
- 8:30Spark Operator pipelines
- 14:32Live Spark demo
- 24:40Spark Operator challenges
- 26:06OCR use case on Spark
- 32:11Summary
related talks
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
24mInteractive Spark at Your Fingertips: Integrating SparkConnect into Kubeflow Noteboo...
26mBuilding Apache Spark Clusters on K8s for AI Workloads
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
29mDynamic Large Scale Spark on Kubernetes
35mBuilding Cross-Cloud ML Pipelines with Kubeflow with Spark & Tensorflow
21mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
34mKick-Ass Tools and Processes for Scalable Data Science