Spark and Flink Running Scalable in Kubernetes #bbuzz
summary
Shows that replacing shared multi-tenant clusters with per-job temporary clusters and an operator-based deployment reduces noisy-neighbor interference but creates startup overhead and orchestration complexity. Demonstrates integration with workflow engines like Airflow, use of per-job container images and templates, and an autoscaling pattern that spawns and destroys clusters on demand. Calls out real pain points: shuffle and network congestion, checkpoint fragility, frequent restarts, lack of centralized logs and costly local-storage constraints.
AI-written summary · about
chapters (9)
- 0:00Introduction and motivation
- 2:24Job characteristics and infrastructure needs
- 7:01Dynamic clusters and auto-scaling
- 12:01Cluster management and workflow integration
- 17:38Cost efficiency and resource optimization
- 23:24Scalability challenges and solutions
- 28:11Orchestration and resource allocation
- 30:14Production experience and lessons learned
- 35:15Summary and Q&A
related talks
30mBeyond Experimental: Spark on Kubernetes
24mDynamic Big Data in the Cloud with Kubernetes
29mDynamic Large Scale Spark on Kubernetes
21mSpark on Kubernetes: The Elastic Story
39mSpark on Kubernetes: Best Practice and Performance
26mBuilding Apache Spark Clusters on K8s for AI Workloads
26mKubernetes & real-time analytics: How to connect these worlds with Apache Flink
17mGetting Started with Flink for Small Teams w/ RabbitMQ and Kubernetes