Real-time analytics with Apache Cassandra & Apache Spark

on YouTube ↗38m

" summary

Demo shows Apache Cassandra for raw time-series storage, Kafka for ingestion and Apache Spark (including the Spark Cassandra connector) for both batch and streaming analytics, collocated on the same nodes to push computation to the data. Materialized views (daily aggregates, highs/lows) use Cassandra counters and stay current with 500 ms micro-batches, avoiding nightly ETL. Notes include topology-aware replication for workload isolation across datacenters and hinted handoff (default retention ~3 hours) for temporarily unavailable replicas.

AI-written summary · about

" chapters (11)

1. 0:00Introduction and setup
2. 3:58Weather data use case
3. 8:13Cassandra overview and Dynamo paper
4. 16:59Cassandra consistency and replication
5. 18:44Spark introduction
6. 23:01Spark components and RDDs
7. 26:30Spark deployment with Cassandra
8. 28:14Cassandra data modeling
9. 31:26Materialized views with Spark
10. 32:50Scala code walkthrough
11. 36:42Combining streaming and batch jobs

" related talks

• 52mSolving Analytical Problems using Apache SparkRohit Bhardwaj · Developer Summit 2020
• 59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & AkkaHelena Edelson · Philly ETE 2015
• 40mSpark Analytics on Cassandra DataTim Berglund · Jfokus 2016
• 54mHarnessing the Power of Spark & Cassandra within your Spring AppSteve Pember · Spring I/O 2017
• 33mTo Spark or Not to SparkRussell Alexander Spitzer · Scale By The Bay 2019
• 30mArchitecting Applications With Multiple Open-Source Big Data TechnologiesPaul Brebner · Community Over Code EU 2024
• 21mModeling Financial Data In Cassandra To Serve Real Time And Batch WorkloadsGokul Prabagaren · ApacheCon@Home 2021
• 50mIntroduction to Real-Time Big Data with Apache SparkTaras Matyashovsky · JavaDay Lviv 2015