Challenges of Spark Application coexisting with NoSQL databases
summary
Capital One describes a rewards platform built with Apache Spark, MongoDB and Cassandra that processes millions of transactions daily and billions yearly; talk summarizes concrete failures and fixes: Cassandra partition key order causes table scans when mismatched, use batching aligned to Spark partitions to avoid hotspots, model duplication becomes unavoidable after MVP so plan data models for query patterns up front, manage MongoDB connections at the JVM/partition level, and beware the MongoSpark connector’s partitioner behavior.
AI-written summary · about
chapters (8)
- 0:00Introduction and context
- 3:19Apache Spark Cassandra MongoDB overview
- 8:32Rewards platform architecture
- 10:17Cassandra key sequence lesson
- 13:28Cassandra partition key design
- 16:43MongoDB schema-less challenges
- 18:53MongoDB indexing importance
- 20:19Speaker background and closing
related talks
41mOptimising Apache Spark and SQL for Improved Performance
40mSpark Analytics on Cassandra Data
57mMulti Tenant NoSQL and NewSQL Cloud data Design Patterns Deep dive
04mInterview
04mKuassi Mensah Interview
50mChallenges that everyone struggles while productionizing Apache Spark workloads
55mSolving Classical Data Analytic Task by Using Modern Distributed Databases
21mLightning fast cluster computing with Cassandra and Spark