Enriching the Data vs Filtering in Spark
summary
Compares two Spark patterns and shows a concrete replacement: swapping expensive in-memory inner join filtering for a per-row enrichment pipeline using left outer join that appends flags and contextual columns. Implements enrichment at each rule step, keeps row-level history for debugging, removes the need for expensive stage counts, and runs at scale — the platform processes billions of transactions and issues millions of rewards with the enriching pattern in production, improving traceability and root-cause analysis without changing final eligibility results.
AI-written summary · about
chapters (9)
- 0:00Spark data processing intro
- 2:39Card Rewards use case
- 4:02Filtering approach explained
- 6:51Challenges with filtering
- 8:41Enriching data approach
- 10:26Enriching data in practice
- 12:35Advantages of enriching
- 14:02Production switch results
- 15:32Closing remarks
related talks
59mSpark 2.0
46mSparking Success: Unveiling the Journey of Apache Spark Application Development
36mData pipelines engineering made simple with Scala
43mIntroduction to Spark Datasets
31mHow to quickly build Data Pipelines for Data Scientists
26mLarge Scale Data Validation (with Spark and Dask)
53mTips and tricks for robust big data applications
18mScalable, Code-Free ETL