Scaling Machine Learning Workflows to Big Data with Fugue
Talkat KubeCon + CloudNativeCon North America 2021 · on Friday, 2021-10-15· at 11:00· 411 Theater + Online·
summary
Fugue lets native Python/Pandas logic run unchanged on Spark or Dask by translating a few calls into engine operations; demos convert a row-wise concat and a sklearn logistic-predict into distributed jobs with transform or a one line of code switch to Spark on Kubernetes. A per-category training pipeline is built with partition/zip/transform and yields about ~10% uplift in precision versus one global model. Boilerplate drops from many extra helper functions to minimal glue and the workflows run end-to-end.
AI-written summary · about
chapters (21)
- 0:00Introduction
- 0:19Demo Overview
- 1:01Han Wang Introduction
- 1:45First Example
- 4:40Spark
- 6:23Transformation
- 12:46Fugue Code
- 15:02Model
- 15:58Field Workflow
- 17:02Results
- 17:18Physical
- 18:02Prediction
- 19:59Pandas vs Spark
- 20:41Lazy evaluation of Spark
- 21:20Partitioning
- 21:55Testing
- 22:22Fugue
- 22:56Decouple logic and execution
- 24:06Demo
- 26:53Notebook extension
- 27:37Recap
related talks
35mPySpark : Combining Machine Learning & Big Data
38mBridging the Chasm Between Research & Software Development
23mRethinking scalable machine learning with Spark ecosystem
54mScaling Machine Learning Pipelines in Cloud
28mRun Your Data and Machine Learning Workflows on Kubernetes with Flyte
29mScaling Python-Powered Machine Learning With Snowflake
35mBringing Your Data Pipeline into The Machine Learning Era
36mScaling data workloads using the best of both worlds: pandas and Spark