Tianqi Chen, Computer Science PhD Student, University of Washington at MLconf Seattle 2017

on YouTube ↗24m

" summary

XGBoost adds monotonic constraints and a new histogram algorithm that gives 10x single-thread speedup and 4–7x multi-thread gains, plus a GPU module that multiplies throughput ~4x. MXNet combines declarative and imperative APIs, a synchronized scheduler that auto-parallelizes GPU/PCIe communication, and scales to 200 GPUs with near–weak-linear scaling. Memory recomputation reduces training memory so MXNet can fit ~1,000-layer ResNet on one Titan X. MVM provides a portable IR and lowers to CUDA, OpenCL, Metal and CPU backends.

AI-written summary · about

" chapters (23)

1. 0:00Intro
2. 0:17Machine Learning Impact us All
3. 0:41Learning System: The Engine of Revol
4. 1:17A Method to Solve Half of the Problem
5. 2:24Monotonic Functions
6. 3:42GPU based Optimization
7. 4:14Modularity: Platform Agnostic Engine
8. 6:08Solution to the other half of problems
9. 6:36MXNet: A Scalable Deep Learning Syst
10. 7:02Declarative vs Imperative Programs • Declarative graphs are easy to store, port, and optimize
11. 8:23MXNet's Approach: Mixed Programmi
12. 9:12Need for Parallelism
13. 9:46Parallel Programs are Painful to Write.
14. 10:17Solution: Auto Parallelization with Dependency Engine
15. 11:38Scaling Up is Good, How about Big Mo
16. 12:30Memory Optimization
17. 12:58Trade Computation for Memory . Do re-computation instead of saving
18. 14:30Deep Learning System will Become more Heterogeneous
19. 15:20Unix Philosophy vs Monolithic System
20. 15:43NNVM: High Level Graph Optimization Deep learning
21. 16:27The Challenge for IR of Deep Learning Syst
22. 18:17Graph, Attribute, and Pass
23. 18:59In Progress: Multi-Level Compilation

" related talks

• 42mAnima Anadkumar, Principal Scientist, Amazon Web Services, Endowed Professor, CalTechAnima Anadkumar · MLconf SF 2017
• 42mAlex Smola, Professor in the Machine Learning Department, Carnegie Mellon UniversityAlex Smola · MLconf SF 2015
• 32mTaming the Machine: Basics of ML Models Training and Inference OptimizationVladimir Osin · PyData Eindhoven 2023
• 30mApache MXNet 2.0: Towards Synergistic ML and DL with StandardizationSheng Zha · ApacheCon@Home 2021
• 29mAccelerate NumPy Data Science Workloads and Deep Learning ApplicationsHao Jin · PyData Los Angeles 2019
• 20mApache MXNet, a highly memory efficient deep learning frameworkGirish Patil · Anthill Inside 2017
• 22mMonoNN: Enabling a New Monolithic Optimization Space for Neural Network Inference...Wei Lin, Haojun Xia, Zhen Zheng, Donglin Zhuang, Shuaiwen Leon Song, Xiafei Qiu, Junjie Bai · 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI '24)
• 01mTianqi Chen, Industry Impact Student Research Award WinnerTianqi Chen · MLconf SF 2016