Deep Learning and NLP with Spark
summary
Implements an LSTM sentiment classifier on Spark using DL4J, ND4J and the Google News word2vec (binary, expands to ~3.5 GB in memory) to convert IMDb reviews into 300‑dim vectors. Trains many‑to‑one LSTMs (sequence length ~100, batch size 50, 12,500 train files per class) with SparkDl4JMultiLayer with SGD and parameter averaging, saves models to Parquet and serves streaming predictions. Demonstrates a Spark Notebook and Juju deploy of Mesos + GPU, notes memory/broadcast limits when loading the large word2vec and ongoing work to distribute that step.
AI-written summary · about
chapters (19)
- 0:00Introduction to NLP and ML
- 0:30Machine learning examples
- 2:24Natural language challenges
- 4:00NLP techniques and applications
- 5:30Understanding neural networks
- 8:35Recurrent neural networks
- 10:44Inputs and outputs in NLP
- 13:02Deep learning for Java
- 14:53Data science notebook demo
- 16:14Cluster deployment with Juju
- 17:38GPU scheduling in Mesos
- 18:24Sentiment analysis workflow
- 25:06Distributed training with Spark
- 27:55Data preparation and processing
- 30:40Iterator and data handling
- 33:28Model training and prediction
- 35:33Limitations and future steps
- 36:24Scala data science resources
- 37:47Q&A and concluding remarks
related talks
51mDeep Learning with DL4J and Apache Spark?
40mScala and Spark in the New Deep Learning Era
35mJVM based DeepLearing on IoT data with Apache Spark
31mFrom Pandas to Spark
40mScaling From Research to Production with Skymind DL4J and ScalNet
43mDetecting fake reviews at scale using Apache Spark and John Snow Labs
23mRethinking scalable machine learning with Spark ecosystem
54mDeep Learning with Apache Spark