Deep distributed decision trees on Apache Spark
summary
Yggdrasil is a Scala package for Apache Spark that trains deep decision trees by column-partitioning instead of row partitioning, keeping features sorted and operating on run-length compressed columns while communicating only sparse bitvectors of left/right decisions. It removes approximate binning, scales roughly linearly where traditional Spark/MLlib grows exponentially with depth, and reports up to 24x faster and an order-of-magnitude speedup on a high-dimensional Yahoo dataset. Available as a Spark 1.6 package; swap DecisionTreeClassifier for Yggdrasil and drop maxBins.
AI-written summary · about
chapters (26)
- 0:00Introduction
- 0:40Whos familiar with decision trees
- 1:26Planet
- 2:55Planet tradeoff
- 4:30Proposed approach
- 5:45Column partitioning
- 6:22Feature partitioning
- 6:55Best feature split
- 7:34Side by side comparison
- 8:47Yggdrasil
- 9:07Partitioning
- 9:21Communication cost
- 10:39Splitting
- 11:04Sorting
- 12:29Why sort
- 13:23Linear sort
- 14:08Optimizations
- 14:41Analytics on compressed data
- 15:02Why compression
- 16:01Results
- 16:31Regression task
- 16:54Communication costs
- 17:15Tree depth
- 18:01Spark compatibility
- 18:18Future plans
- 18:36Quality
related talks
53mA Taste of Random Decision Forests on Apache Spark
39mMachine Learning with Scala on Spark
50mApache Spark vs Machine Learning
54mDeep Learning with Apache Spark
48mMachine Learning in Azure Databricks
42mScalable Data Science in Python and R on Apache Spark
44mData Science at Scale with Apache Spark
24mJoseph K. Bradley, Software Engineer, Databricks Inc. @ MLconf SEA