Detecting fake reviews at scale using Apache Spark and John Snow Labs
summary
End-to-end NLP pipeline claims to detect fake reviews at scale by ingesting ~1,600 labeled hotel reviews, converting raw text into Delta tables, running Spark NLP (John Snow Labs models) for tokenization, lemmatization and CountVectorizer, training a logistic regression in a Spark ML pipeline, and logging runs and an F1 score to MLflow. The demo shows query time dropping from ~20s to much faster after moving CSV to Delta Lake on the Databricks Lakehouse, registers the model, and exposes it as a REST endpoint.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker background
- 1:06Understanding spam reviews and business impact
- 3:57Steps in building spam detection system
- 5:19Databricks lakehouse platform overview
- 8:28Demo setup and prerequisites
- 13:50Data ingestion and Delta Lake setup
- 19:07Feature engineering with Spark NLP
- 27:42Training classification model with MLflow
- 32:01Model evaluation and logging
- 39:56Productionizing and model serving
- 42:47Summary and Q&A
related talks
34mActionable Insights with Real-time Streaming Analytics of Customer Reviews
21mFrom 0 to a crowd sourced neural network in a jiffy
50mDeep Learning and NLP with Spark
46mProcessing Hotel Reviews with Python
25mReal-Time Fraud Detection at Scale
29mBuilding End to End Gen AI Powered Analytics: From Data to Production
45mDeep Learning and NLP with Spark
01h 22mWhen Rotten Tomatoes Isn’t Enough: Analyzing Twitter Movie Reviews