Building a modern data platform on Amazon EKS
summary
Covers using Amazon EKS to run and scale analytics and ML workloads such as Spark jobs, ML processes, and workflow managers. Presents the open-source Data On Amazon EKS project, its IaC templates, deployment examples, and architectural patterns for provisioning, monitoring, and efficient scaling of high-scale data platforms.
AI-written summary · about
chapters (22)
- 0:00Intro
- 1:40Why running data workloads on Kubernetes?
- 7:05EKS blueprints - concepts
- 7:55What is data on EKS?
- 9:49Apache Spark is popular
- 10:40Spark selection considerations
- 11:57Spark challenge-scheduling
- 14:14Spark-scheduling blueprints
- 16:03Spark-disk storage blueprints
- 18:27Spark-shared storage blueprints
- 18:51Spark-compute blueprints
- 19:49Spark-observability blueprints
- 20:59Bootstrap data on EKS-spark operator
- 22:44Why machine learning with containers?
- 24:41Why machine learning on Kubernetes?
- 25:29Kubernetes ML training
- 26:35Kubernetes ML Inference
- 27:05Kubernetes for ML-ML challenges
- 29:29Kubeflow pipelines overview
- 30:15Code example - creating a pipeline
- 31:32Kubeflow on Amazon EKS
- 39:22Takeaways
related talks
14mBuilding a high-performance analytics ecosystem on Amazon EKS
20mBuilding modern applications using Amazon Elastic Kubernetes Service
34mKick-Ass Tools and Processes for Scalable Data Science
17mKeynote: Kubernetes: Lessons from the Trenches
26mRe-architecting the Data Platform with Spark on Kubernetes
53mHow to build a modern data architecture
13mDesigning an Autonomous Workbench for Data Science on AWS
49mBuilding an Enterprise Data Lake Architecture on AWS