Efficient Integration of Kubeflow Tools: An ISS Data Case Study on Anomaly Detection
Talkat Kubeflow Summit Europe 2024 · on Tuesday, 2024-03-19· at 15:15· Pavilion 7 | Level 7.3 | Room S02·
summary
Kubeflow Pipelines v2 orchestrates Dask for parallel preprocessing, Katib for hyperparameter search, PyTorch Operator for multi‑node training and KServe for deployment inside a single workflow, but it requires concrete hacks: the project builds one image with multiple entry points managed by Poetry and Click, extracts MinIO artifact paths from KFP inputs and uses MinIO credential injection into spawned pods, and runs distributed training on four GPUs with PyTorch DDP. Open‑source KFP backend limits some parallel features; local runs speed debugging. Code and paper are on GitHub.
AI-written summary · about
chapters (10)
- 0:00ISS anomaly detection project
- 2:27Use case overview
- 4:56Kubeflow pipeline components
- 8:14Single Docker image with multiple entry points
- 11:43Local debugging of components
- 12:05Kubernetes services from lightweight components
- 14:14Data access challenges
- 16:21MinIO data access workaround
- 17:46Key takeaways
- 18:58Q&A
related talks
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
21mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
51mMachine Learning using Kubeflow and Kubernetes
36mIntroducing Kubeflow (w. Special Guests Tensorflow and Apache Spark)
36mStreamlining ML Workflows With the Unified Kubeflow SDK
31mKubeflow - Data Science on Steroids
39mOperating Deep Learning Pipelines Anywhere Using Kubeflow
54mScaling Machine Learning Pipelines in Cloud