Spotting trends and tailoring recommendations: PySpark on Big Data in fashion
summary
Mallzee builds a per-user recommender that runs on a PySpark cluster, featurizing products with scraped metadata, keywords and 14 hand-tagged brand style dimensions, collapsing those into cosine-similarity style profiles and type/color/keyword save ratios. Training is a weekly pipeline that fits random forest per-user models to predict saves on thousands of new items. Validation shows high cross‑validation scores for “good” users but strong class imbalance: easy to predict hides, harder to predict likes, and scaling remains the main engineering challenge.
AI-written summary · about
chapters (15)
- 0:00Fashion recommendation intro
- 1:45User interactions and product data
- 4:36NLP for product features
- 5:21Brand style profiles
- 9:02User style profiles from brands
- 12:17Brand similarity with cosine
- 13:40User ratios: type, color, keywords
- 14:46Training sets for personalization
- 15:54Validating recommendation accuracy
- 17:23Data imbalance and user variability
- 18:32Trendy users and fashion trends
- 19:56Scaling recommendations efficiently
- 21:00PySpark for scalable processing
- 22:04Recommendation pipeline
- 23:10Q&A
related talks
26mLarge-Scale Recommendation System with Python and Spark
42mAn Algorithm of Style
32mLeveraging recommender systems to personalise search results
29mThe Challenges of Building a Personalized Recommendation System in E-Commerce
28mThe Power of Personalized Customer Experiences Through AI Innovations
31mCreating recommender systems with Spark, Scala and Prediction.Io
35mLife, Death, and Shopping
40mIntroduction to Machine Learning