Shparkley: Scaling Shapley values with Spark
summary
Implementation of Shapley IME in Spark scales exact Monte Carlo Shapley approximations to large training sets by broadcasting the model to executors, partitioning the background data, and generating counterfactual pairs per permutation locally. It produces tall Spark dataframes of per-feature marginal contributions, supports training weights, reuses rows to batch predictions for speed, and preserves efficiency (sum plus mean equals prediction). On a 30k-row demo with 8 partitions it finishes in ~50s vs kernel SHAP ~1m20s and yields Shapley values close to existing implementations.
AI-written summary · about
chapters (8)
- 0:00Start
- 2:52What are we looking for?
- 5:44Four feature model example
- 6:56Permutations
- 8:13Comparing performance
- 9:21Monte-Carlo approximation
- 16:05Runtime and convergence
- 24:03Conclusion
related talks
27mInterpretability and explainability in machine learning
44mWhat is Your Data Worth? Equitable Data Valuation in Machine Learning
24mTaming Data Skew in Production ML Pipelines
17mMachine Learning at the Speed of Light with Spark
34mRethinking Feature Importance-Evaluating SHAP & TreeSHAP for Tree-Based MLM
44mData Science at Scale with Apache Spark
39mMachine Learning with Scala on Spark
35mTaking a peek under the hood - Interpreting black box model