Real-time Data Processing with Apache Spark Structured Streaming

on YouTube ↗44m

" summary

Apache Spark Structured Streaming is presented as a scalable, simple API for real-time-style processing, transformation and aggregation of large transaction volumes. The talk explains using Structured Streaming to process over 100 million tariff-attempt transactions per day, and describes the processing architecture and stream-related considerations useful for transforming large datasets.

AI-written summary · about

" chapters (13)

1. 0:00Start
2. 2:57Introduction to Apache Spark
3. 6:11Apache Spark Structured Stream
4. 13:08Output Modes, data output
5. 14:02Types of Output Sinks
6. 15:52Fault Tolerance with Checkpoints
7. 17:00Handling Out-of-Order Data
8. 26:00Processing (Spark SQL API) v1
9. 28:33Processing (Spark SQL) v2
10. 29:50Output Data Source
11. 32:47RESULTS OBTAINED
12. 35:02EVOLUTION
13. 37:04Performance Check

" related talks

• 01h 01mApache Spark Structured Stream: A Near-Real-Time Data GrinderEiti Kimura · GDG DataFest 2018
• 42mBuilding Robust Streaming Data Pipelines with Apache SparkZak Hassan · Open Source Summit North America 2017
• 45mFrom Big Data to Fast Data: Apache SparkStefano Baghino · Codemotion Milan 2015
• 44mApache Spark on Azure - solution (...)Kamil Dworak · 4Developers Katowice 2019
• 52mSolving Analytical Problems using Apache SparkRohit Bhardwaj · Developer Summit 2020
• 20mStreaming ArchitecturesJuan Camilo Salazar · DevOpsDays Bogotá 2023
• 08mReal-Time Data Systems: Speed, Scale, and ResilienceLakshmi Bhargavi Mullapudi · Conf42 JavaScript 2025
• 41mStreaming: an Alternative to CRUD & Batch ProcessingSamuel Roze · AFUP Day 2020 Lille