Riak, Redis, Apache Solr & Spark: Deploying with the Basho Data Platform
summary
A deployable platform that wires Riak (default 3x replication, 64 partitions), Redis and Solr into Spark jobs is described and demoed: a Redis proxy in C provides a pull-through cache, Riak gets data types to resolve conflicts, and Solr indexes are kept synced via active anti-entropy. Service manager implements a Paxos-like leader election for Spark metadata. The pipeline auto-heals missing objects and indexes, supports multi-cluster replication, and is reported as operational at customer scale.
AI-written summary · about
chapters (25)
- 0:00Intro
- 1:48Best Buy
- 2:31Who uses Riak
- 5:07Success and failure
- 5:47Scale
- 8:53Mobile Client
- 10:16Data Search
- 11:26Growing Pants
- 14:17Scale Availability
- 15:29Riak Architecture
- 17:41Variability Consistency
- 21:38Cluster Reputation
- 23:33Search Scale
- 25:13Improved Search
- 25:24Middleware
- 27:07Redis
- 33:43Why Redis with Data Platform
- 35:38Service Manager
- 36:44Cluster Configuration
- 37:14Lua Scripting
- 38:46Spark Mind
- 39:06High Availability
- 40:02Metadata
- 40:27Match by website
- 41:40Any questions
related talks
16mLogging Apache Spark - How we made it easy
50mDesigning a scalable data platform based on Apache Spark/Iceberg
42mApache Zeppelin
01h 18mScalable Data Science with Spark and R
55mSolving Classical Data Analytic Task by Using Modern Distributed Databases
28mAn introduction to Riak
21mChallenges of Spark Application coexisting with NoSQL databases
26mFrom Spark to Elasticsearch and Back