intro video
summary
Explains the incremental ETL claim as a pattern that processes only changed data so pipelines run faster and cheaper, and shows it implemented on Databricks using Apache Spark, Delta Lake and MLflow. Describes how checkpoints, change data capture or file-based diffs are used, how state is stored and reconciled, and what latency and cost tradeoffs look like for continuous versus batch ingestion. Demonstrates concrete engineering choices and operational outcomes.
AI-written summary · about
related talks
32mBig Data Processing : Where do we go next ?
02h 18mDistributed Systems: In the Daily Life of the Data Engineer
34mA journey of scaling high-volume low-latency stateful processing
25mSPARK AND AKKA FOR BIG DATA SYSTEMS, IN PRACTICE
28mA look at Flink’s internal data structures & algorithms for efficient checkpointing
24mUnder Pressure: Applying ML in Real Time
14mManaging decisions with Amazon Kinesis for better outcomes
44mThe Art and Benefits of Computing Eventfully