Apache Spark Declarative Pipelines in Action
summary
Spark 4.1 Declarative Pipelines replace imperative glue code with declarative configs and runs pipelines that auto-handle validation, dependencies, retries and parallelism. The demo wires a 3-line streaming tables ingestion to a SQL materialized views aggregation over live ADS‑B data from tens of thousands of aircraft using an open‑sourced OpenSky data source, built in under 30 minutes; the managed demo ingests ~3,500 records and can run continuously, scheduled, or in real‑time (sub‑second) mode.
AI-written summary · about
chapters (8)
- 0:00Introduction and motivation
- 7:21Why declarative pipelines
- 14:45Streaming tables and materialized views
- 22:30Live demo of flights pipeline
- 33:01Managed service and governance
- 35:47AI-assisted pipeline generation
- 37:14Real-time mode and advanced use cases
- 40:01Q&A on batch vs streaming
related talks
42mBuilding Robust Streaming Data Pipelines with Apache Spark
41mSpeeding Up Data Pipeline Construction in Apache Spark
14mStreaming Data Pipelines to Enhance Real-Time Decision-Making
01h 20mSupercharge Your Real-Time Analytics: Snowpipe Streaming & Dynamic Tables in Snowflake
31mOne Data Pipeline to Rule Them All
52mSolving Analytical Problems using Apache Spark
47mBuilding Modern Data Streaming Apps with Open Source
44mApache Spark on Azure - solution (...)