Streaming Data Pipelines on Apache Mesos: Lessons Learned
summary
Argues that Mesos outperforms YARN for mixed streaming and microservice workloads, showing concrete reasons: Mesos two-level scheduling lets frameworks decide task packing, Mesos supports fine-grained resources including GPUs, and container isolation enables long-running daemons and short-lived jobs side-by-side. Describes using Kafka + Spark + Flink + Akka Streams on DC/OS, tradeoffs (Spark mini-batches vs Flink true low-latency), problems encountered (Spark checkpoint bugs, state recovery), and runtime solutions like dynamic resource offers.
AI-written summary · about
chapters (32)
- 0:00Intro
- 1:45Why do we care
- 2:55Should we use Hadoop
- 5:25Whats wrong with Hadoop
- 6:30Whats not to like
- 6:47Yarn
- 8:26Container Support
- 9:04Streaming Services
- 9:22Microservices
- 10:45Why Mesos
- 14:12Spark
- 15:10Streaming
- 15:28Continuous Processing
- 16:31Lifespan
- 18:05Scalability
- 18:37Checkpointing
- 19:39Dynamic Resource Allocation
- 21:46Latency Budget
- 23:35Machine Learning
- 25:47Interoperability
- 26:17Tools
- 26:29Kafka streams
- 28:40Akka streams
- 29:58Embedding Kafka and Akka streams
- 31:07Spark Streaming
- 32:55Flink
- 36:43Microservices and Fast Data
- 37:14Microservice Architecture
- 37:39Streaming Architectures
- 38:04Microservices vs Big Data
- 38:46Common Mechanisms
- 39:14Wheres the Beer
related talks
39mDeploying Fast Data Pipelines
42mApache Flink meets Apache Mesos and DC/OS
35mBringing Your Data Pipeline into The Machine Learning Era
59mBuilding Distributed Systems with Apache Mesos
42mBuilding Robust Streaming Data Pipelines with Apache Spark
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
39mThe Sparkling Solution that Scales to IoT