Streaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
summary
Shows a runnable Scala Akka application that wires Kafka producers, Spark Streaming micro-batches, the Spark–Cassandra connector and Cassandra tables to implement a Lambda-style pipeline with exactly-once guarantees and incremental model updates via Spark ML (online SGD). Demonstrates collocating Spark and Cassandra nodes to leverage co-located Spark+Cassandra for data locality, time-series schemas that use clustering keys and time UUIDs to avoid overwrites, and using Kafka topics for backpressure and durable replay. Includes code, a weather time-series demo, and operational fault-tolerance notes.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 3:32Real-time analytics challenges
- 7:07Overview of technologies
- 13:30Architecture and integration
- 21:11Spark Streaming basics
- 26:24Cassandra features and use cases
- 33:51Spark Cassandra connector
- 40:32Kafka streaming examples
- 44:03Spark ML integration
- 45:48Sample application overview
- 58:56Q&A and resources
related talks
38mReal-time analytics with Apache Cassandra & Apache Spark
52mSolving Analytical Problems using Apache Spark
42mBuilding Robust Streaming Data Pipelines with Apache Spark
30mArchitecting Applications With Multiple Open-Source Big Data Technologies
40mSpark Analytics on Cassandra Data
53mApache Kafka and KSQL in Action : Lets Build a Streaming Data Pipeline!
54mHarnessing the Power of Spark & Cassandra within your Spring App
50mIntroduction to Apache Kafka