Deploying Fast Data Pipelines
summary
Overview of deploying Fast Data on Mesos/DC/OS with concrete choices and tradeoffs: uses Kafka + Spark Streaming + Elasticsearch and optionally Flink, Cassandra and Marathon; explains micro-batch vs native streaming, checkpointing frequency, and exactly-once vs at-least-once delivery costs; shows a 11-node cluster running a taxi-tracking demo (simulated NYC taxis) that scales, replays time windows, builds heatmaps and serves queries, and notes the live demo is video-backed but the deployment artifacts run successfully.
AI-written summary · about
chapters (11)
- 0:00Start
- 1:39WEB SCALE? A380 SCALE
- 2:32HYPERSCALE MEANS VOLUME AND VELOCITY
- 4:10The Fast Data Universe
- 9:24DELIVERY GUARANTEES
- 11:13STREAM PROCESSING
- 12:34choose?
- 17:40APACHE SPARK STREAMING
- 18:43STORAGE
- 20:05APACHE CASSANDRA
- 25:13SERVICE OPERATIONS
related talks
42mStreaming Data Pipelines on Apache Mesos: Lessons Learned
42mBuilding Robust Streaming Data Pipelines with Apache Spark
35mBringing Your Data Pipeline into The Machine Learning Era
40mFrom concept to production; the fastest way to develop & deploy Apache Flink jobs
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
12mGetting started with fast data architecture
43mA Data Streaming Architecture with Apache Flink #bbuzz
47mBuilding Modern Data Streaming Apps with Open Source