Deploy end to end ML pipeline using Apache spark ...
summary
Deploys a full streaming ML pipeline that uses Apache Spark Structured Streaming, Kubernetes and Kafka to ingest simulated tweets, batch them for throughput, and call an IBM MAX BERT-based sentiment model packaged as a Docker microservice. Demo generates a controllable rate of rows/sec via Spark rate source, scales executors and service replicas, and prints per-airline sentiment aggregates. Code and deployment manifests live in the GitHub repo (ScrapCodes/SS-on-kube).
AI-written summary · about
chapters (19)
- 0:00Start
- 2:04The Machine Learning Workflow
- 2:17The workflow spans teams
- 4:35Machine Learning Deployment
- 5:10What is a "model"?
- 6:09Deep learning pipeline!
- 7:09Containers for ML Deployment
- 8:20The Model Asset Exchange
- 10:19Deployable asset on MAX
- 11:07Spark Streaming helps with the workflow
- 14:21Downside.
- 15:00Introducing Apache spark structured streaming.
- 15:11What is Spark Structured Streaming?
- 16:46Demo: Tweet stream sentiment analysis.
- 17:29A bit about the dataset.
- 18:13MAX Text Sentiment Classifier
- 24:58Single click deployment.
- 27:13Behind the scene of a simplest demo!
- 27:23In a glimpse, first create cluster config.
related talks
33mDeploying and Scaling Spark ML and Tensorflow AI Models
35mSimplification of Data Pipelines by Containerization
23mRethinking scalable machine learning with Spark ecosystem
42mBuilding Robust Streaming Data Pipelines with Apache Spark
01h 47mRadanalytics.io streaming applications workshop
01h 25mApache Spark for Big Data Processing
26mOptimal Approaches for Real-Time Machine Learning with Apache Spark on Kubernetes.
52mMachine Learning Data Pipelines