NLP text recommender system journey to automated training pipeline with Spark and Sagemaker

on YouTube ↗31m

" summary

Built an enterprise NLP recommender that reduces agent time‑to‑resolution by surfacing top‑k articles, using Scala/Python, Apache Spark, Docker and SageMaker + Spark training for automated hyperparameter tuning and CI/CD deployments with rollback. Pipeline prunes a corpus from ~100k articles to ~1k candidates then pairwise ranks with ROC AUC / hit@k checks; supports multiple languages, observability and hundreds of tenant models with periodic retraining and cold‑start default models to avoid data sparsity. Serving latency kept low and deployments are containerized.

AI-written summary · about

" chapters (16)

1. 0:00Start
2. 0:41Agenda
3. 1:26Goal
4. 2:29The Business Metrics
5. 13:40Metrics Ui
6. 15:18The Training Pipeline
7. 17:36Hyper Parameter Tuning
8. 18:03Sample Size
9. 20:15Data Prep
10. 21:10Nlp Feature Generation
11. 23:20Model Deployment
12. 25:51Challenges and the Takeaways
13. 26:52Model Management
14. 28:41Cold Start Problem
15. 29:23The Takeaways
16. 29:41Model Interpretability

" related talks

• 48mBuild a Recommender System in Apache Spark and Integrate It Using AkkaWillem Meints · Scala Days Berlin 2016
• 55mBuilding a Recommendation Engine with Machine Learning TechniquesBrian Sam-Bodden · Full Stack Fest 2016
• 31mCreating recommender systems with Spark, Scala and Prediction.IoFlorian Krause · code.talks 2017
• 26mFrom Spark to Elasticsearch and BackSonya Liberman · Big Data Conference Europe 2019
• 26mLarge-Scale Recommendation System with Python and SparkPhil Anderson · PyOhio 2018
• 35mIntegrating a Spark ML Pipeline into a High-Load Low-Latency Recommendation ServiceIlya Malov · Joker 2023
• 37mBuild a Recommender System in Apache Spark and Integrate it Using AkkaWillem Meints · Scala Days Berlin 2016
• 32mResurrecting a Recommendations PlatformLeemay Nassery · Strange Loop 2018