Alex Smola, Professor in the Machine Learning Department, Carnegie Mellon University
summary
Distributed key‑value Parameter Server coordinator plus careful single‑machine caching and partitioning let large models run fast: 100 billion variables and data points train in under an hour on the cluster, and a Netflix‑scale matrix‑factorization runs ~8× faster than a naive implementation and achieves 80× speedup across 16 machines. A new deep stack built in MXNet runs Inception/ImageNet training on spot AWS GPUs — synchronous minibatch 512 — in about 4 hours for ~$24, with lower memory use than TensorFlow, and attention‑based visual question answering models beat prior published results.
AI-written summary · about
chapters (18)
- 0:00Start
- 1:18Multicore
- 1:38GPUs (for Deep Learning)
- 5:54Parameter Server Template
- 6:44Solving it at scale
- 7:59Recommender Systems
- 12:13Key Ideas
- 13:54Speed (c4.8xlarge)
- 14:24Convergence
- 15:30Factorization Machines
- 17:51Better Models
- 18:37Faster Solver (small Criteo)
- 18:59Multiple Machines
- 27:14Side-effects of using MXNET
- 29:08Tensorflow vs. MXNET
- 34:23Memory Networks (Sukhbataar, Szlam, Fergus, Weston, 2015)
- 35:15Memory networks for images
- 38:44Summary
related talks
42mAnima Anadkumar, Principal Scientist, Amazon Web Services, Endowed Professor, CalTech
01h 16mDeep Dive into Deep Learning with MXNet
18mModel Serving for Deep Learning
36mAccelerating High-Performance Machine Learning at Scale i...
44mScaling ML from 0 to millions of users
32mDeep Learning with TensorFlow, PyTorch and MXNet on AWS
29mAccelerate NumPy Data Science Workloads and Deep Learning Applications
33mHigh Performance Deep Learning on Edge Devices With