Sparking our way to YugabyteDB: Our journey from ScyllaDB to
summary
Helpshift migrates end‑user data from ScyllaDB to YugabyteDB using a dual‑write strategy and a Spark‑based migrator on AWS EMR that reads Cassandra, transforms rows to normalized Postgres schemas, and pre‑splits Yugabyte tablets. A memory spike and node failure expose lock contention from mixed insert/update transactions; separating updates and inserts, replacing Odyssey pooling with Yugabyte’s cluster‑aware driver, rate limiting, and retries stabilise the run. Final migration hits about 45k ops/s and completes in 12 hours, after which Yugabyte becomes primary.
AI-written summary · about
chapters (15)
- 0:00Introduction and speaker
- 1:11Helpshift and YugabyteDB
- 5:44Migration challenges
- 7:49Why YugabyteDB
- 9:44Migration strategy
- 12:31Data migration architecture
- 15:23Safety and best practices
- 17:06Benchmarking and tuning
- 18:01Production readiness
- 20:08Initial learnings and issues
- 21:01Identifying the root cause
- 22:17Optimizing the migration job
- 23:22Final production success
- 25:17Migration results
- 26:15Future roadmap
related talks
29mHow Helpshift Accelerated Growth by Migrating to
20mDatabase migration: Insights on Dual Read Write Strategy
35mA Customer’s Journey: Migrating from NoSQL to a Cloud Native Distributed SQL Database
28mDisrupting the Status Quo with Distributed SQL to Deliver Data-Driven Solutions
13mStrategic Shift to YugabyteDB as Core Database
36m5 Hard-Learned Lessons from Successful Database Migration and Modernization
19mEvolve: A Database Journey from Ground to Cloud
26mEfforts Toward Adoption and Uncovering Hidden Features