SPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
summary
Compares concrete tradeoffs: use Akka Cluster and actors for low latency stream or custom graph processing with built in back pressure via reactive drivers and easy HTTP/JMS integration; use Spark for HDFS landings, massive parallel ingestion via partitioned import, high throughput batch transforms and Spark ML for batch machine learning. Avoid vendor glue like Sqoop/Flume when you can write typed ingestors. Orchestrate with a clustered coordinator for HA and handle split brain explicitly. Serve small extracts over HTTP, push huge exports through Spark.
AI-written summary · about
chapters (20)
- 0:00Akka vs Spark in Big Data systems
- 0:43DISCLAIMER
- 0:57What is Spark?
- 1:31What is Akka?
- 2:08What is Big Data
- 4:24Data Ingestion
- 5:32Configured vs Programmatic Ingestion
- 6:35Example: Flume flow configuration
- 6:48Akka vs Spark: Sources
- 7:54Akka vs Spark Reactive vs Partitioned
- 10:47Data Transformations: Exploratory Analysis
- 12:15Data Transformations: Akka Streams vs Spark
- 13:40Data Transformations: Akka Actors / Cluster
- 15:21Data Transformations: Machine Learning
- 16:57Process Orchestration & Management
- 18:27Process Orchestration: Ad-hoc mode
- 19:39Process Orchestration: Coordinated mode
- 21:36Extraction: Things to consider
- 23:29Fast Data Stack
- 23:58Big Data Stack
related talks
26mHow to Pick Your Next Streaming Architecture
02h 18mDistributed Systems: In the Daily Life of the Data Engineer
04mEvan Chan Interview
01h 20mAkka Streams for Large Scale Data Processing
42mStreaming Data Pipelines on Apache Mesos: Lessons Learned
19mKeynote Panel Discussion: SMACK in the Enterprise
32mBig Data Processing : Where do we go next ?
05mVladimir Bacvanski Interview