Maja Kabiljo, Software Engineer, Facebook Inc.
summary
A scalable matrix‑factorization pipeline on Apache Giraph implements rotational distribution of item shards to avoid sending feature vectors over every edge, uses a hybrid SGD/ALS decision by item degree, and adds social regularization and fast top‑k extraction. It runs on clusters that handle over a billion users and tens of millions of items, scales to 100+ billion ratings, and reports up to 10x speedups versus Spark MLlib on Amazon review benchmarks while maintaining comparable RMSE.
AI-written summary · about
chapters (15)
- 0:00Intro
- 0:39Motivation and challenges
- 1:18Collaborative Filtering
- 2:11Matrix factorization
- 4:29Training/testing metrics example
- 5:08What is Apache Giraph?
- 7:41Common approach
- 9:19Our solution - rotational approach
- 11:15Comparison with Spark MLlib
- 12:01Hybrid - common + rotational
- 13:41Rotating items
- 16:54Additional features
- 17:53Item similarities
- 19:09Applications
- 20:28Conclusion
related talks
24mJoseph K. Bradley, Software Engineer, Databricks Inc. @ MLconf SEA
24mLarge Scale Graph Computation with Apache Giraph
24mAshfaq Munshi, ML7 Fellow, Pepperdata
40mBuilding your own Distributed Streaming Recommendation Engine
37mBuild a Recommender System in Apache Spark and Integrate it Using Akka
16mAndrew Musselman, Committer & PMC Member, Apache Mahout at MLconf Seattle 2017
17mCarlos Guestrin, CEO of Dato Inc. @ MLconf SEA
45mReal-time, Streaming Advanced Analytics....