Enriching datasets using Apache Spark solves tracing in financial industry

on YouTube ↗23m

" summary

Compares two Apache Spark data-joining strategies—enriching datasets via left outer joins versus filtering via inner joins—applied to credit card transaction processing at Capital One. Describes how moving from filtering to enriching enabled end-to-end, granular tracing required by financial regulation and outlines practical benefits and production issues observed when adopting the enriching approach.

AI-written summary · about

" related talks

• 03mFred Reiss InterviewAlexy Khrabrov, Frederick Reiss · Scale By the Bay 2017
• 21mModeling Financial Data In Cassandra To Serve Real Time And Batch WorkloadsGokul Prabagaren · ApacheCon@Home 2021
• 04mInterviewJianneng Li · Scale By The Bay 2018
• 33mTo Spark or Not to SparkRussell Alexander Spitzer · Scale By The Bay 2019
• 43mOptimizing Spark applications from simple to complex with examplesAleksandr Markachev · HighLoad++ 2024
• 26mReal time Platform for Second Look Business Use Case Using Spark & KafkaIvy Lu · EuroPython 2019
• 36mBig Data Modeling with Spark SQL: Make data valuableJayesh Patel · Data Con LA 2019
• 57mEmergence of Real-Time Analytics: Real-time Analysis of...S. Palthepu · Philly ETE 2016