Building Robust Streaming Data Pipelines with Apache Spark
summary
A working order pipeline demonstrates Apache Camel ingesting HTTP orders, persisting to Cassandra+Kafka+Spark pipeline, and Spark Structured Streaming consuming Kafka, applying watermarking and checkpointing, then writing compressed Parquet files to HDFS. The stack runs in Docker on OpenShift/Kubernetes with dynamic Spark clusters created per job. A Jupyter notebook queries live aggregates as events stream (demo shows counts rising to double digits), and Avro schemas plus Parquet tooling are used for versioned, queryable storage.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:36Data Preparation
- 1:49Data Types
- 2:51Camel
- 8:31Data formats
- 13:51Demo
- 19:17Hive vs Spark
- 26:40Demo Time
- 29:10Demo Starts
- 30:50Logs
- 32:15HDFS
- 34:50Python
- 38:20Code
- 39:50Recap
- 40:50Office Hours
related talks
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
31mOne Data Pipeline to Rule Them All
41mSpeeding Up Data Pipeline Construction in Apache Spark
42mApache Spark Declarative Pipelines in Action
45mData pipelines from zero to solid
39mDeploying Fast Data Pipelines
52mSolving Analytical Problems using Apache Spark