Deep Learning with DL4J and Apache Spark?
summary
Demonstrates distributed deep-learning with DL4J, ND4J and Apache Spark, diagnosing deployment failures on standalone, YARN and Kubernetes clusters such as off-heap memory spikes, Spark job restarts on task failure, Kryo serialization mismatches, and slow image I/O. Presents concrete fixes: use ND4J memory workspaces or tuned GC, add the nd4j-kryo registrator, run the DL4J dashboard as a separate process, store Spark data in serialized formats, and use DL4J batch files for images; all validated on CNN image tasks and production pipelines.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:04Distributed deep learning challenges
- 7:25Deep learning myths vs apps
- 10:15Spark model challenges
- 13:26DL4J features
- 17:45Why DL4J, Spark, ND4J
- 22:01Cluster best practices
- 23:27DL4J memory on Spark
- 27:20Training dashboards
- 30:54Serialization fixes
- 34:09CNN memory on Spark
- 37:21CNN data prep
- 41:19Image preprocessing
- 44:57CNN training on Spark
- 46:34Q&A
related talks
40mScaling From Research to Production with Skymind DL4J and ScalNet
50mBuilding Deep Neural Networks using Deep Learning for Java(DL4J)
45mDeep Learning and NLP with Spark
44mDeep Learning in Production
40mScala and Spark in the New Deep Learning Era
54mDeep Learning with Apache Spark
44mDeep Learning with DeepLearning4J and Spring Boot
04mYuhao Yang Interview