Modern Big Data Pipelines over Kubernetes [I]
summary
Claims Kubernetes replaces YARN as a single scheduler for Spark Presto TensorFlow Kafka Cassandra and Elasticsearch, and shows a live demo: create namespace, helm install services, build and push a function with a CLI, stream IoT car events to a real‑time dashboard while a Zeppelin Spark job reads the same data concurrently. Describes shared memory acceleration via a v3l daemon (or FUSE flex volume) to avoid thousands of TCP connections, and promotes nucleo function service and helm charts as open source for one‑command deployment.
AI-written summary · about
chapters (14)
- 0:00Intro
- 0:31The Big Data Era
- 3:05Cloud Era
- 4:56Recap
- 6:38Data
- 9:19Functions
- 11:48Containers
- 13:58Deployments
- 15:12Current Pipeline
- 16:16Demo
- 20:38Deploy Functions
- 22:43Data Streaming
- 24:38Spark Job
- 30:01Tips
related talks
24mDynamic Big Data in the Cloud with Kubernetes
26mRe-architecting the Data Platform with Spark on Kubernetes
26mBuilding Apache Spark Clusters on K8s for AI Workloads
29mDynamic Large Scale Spark on Kubernetes
34mTransforming Data Processing with Kubernetes: Journey Towards a Self-Serve Data Mesh
22mSPOK - Managing ML/Big Data Spark Workloads at scale on Kubernetes
38mChallenges of and Solutions for Migrating Spark From Legacy Hadoop Clusters To Kubernetes
28mProgress For Big Data In Kubernetes