Data Migration With Confidence

on YouTube ↗37m

" summary

Migration moves millions of rows and 100k photos into Rails while keeping the site live by combining batch LOAD DATA INFILE imports, ActiveRecord-import, and a lazy per-login migration that migrates infrequently used records on demand; dumps get edited with a Ruby streaming script to throttle inserts, images use precomputed image hashes and are uploaded beforehand (ten-day bulk run), passwords fall back to legacy MD5 then re-hash on first login, retries and progress reporting run via job queues, Redis caching and reduced worker count solve IO limits, and the approach proves workable in production.

AI-written summary · about

" chapters (48)

1. 0:00Intro
2. 1:20Data Migration?
3. 1:43Transfer data from System, to System
4. 2:08Data Migration to Existing System
5. 2:24Why Migration?
6. 2:55Get All Data to our system
7. 3:12Provider API Data Dump
8. 3:55Start with a rake task
9. 6:16Modeling Database
10. 6:40Map data to your current system
11. 7:31Setup Test Suite
12. 8:08Better code through boring tests
13. 8:34Use all methods that raises exception
14. 8:46Fail Fast to find all errors
15. 10:10Data Integrity
16. 10:28Idempotent Operation
17. 10:42Run Migration many times
18. 11:25Accuracy
19. 11:46Example Check users with most Recipes
20. 12:58Add more Checker objects
21. 13:14Objects Everywhere
22. 14:16Log Every Unexpected Error
23. 14:48Fix every error you can before real migration
24. 14:58Retry mechanism
25. 15:22MySQL deadlock
26. 16:59Status Reporting
27. 17:52Monitoring CPU Usage
28. 18:17Performance is a Rabbit hole
29. 18:39Preload associations
30. 18:56Avoid unnecessary callbacks
31. 19:27Process multiple records in one job
32. 20:01Migrate Important things first
33. 20:34Scale up the Database
34. 21:12Every change to make it fast
35. 21:35Update all necessities
36. 22:28Stories
37. 23:42Migrate Millions of records
38. 24:31Migrate their data in high priority
39. 25:11Migrate 100K photos
40. 26:10Set the designated hash during migration instead of upload, generate hash
41. 26:20Benchmark how long to finish all
42. 26:3599% photos won't change
43. 26:48Migrate users password to secure auth
44. 27:13System's password auth will fail
45. 27:28Fallback to Legacy Auth
46. 27:44When password matched from legacy auth
47. 28:07Migration done in Ghost Table fashion
48. 28:21Data Dump & Generic Migration Code

" related talks

• 37mData Migration With ConfidenceJuanito Fatas · RedDotRubyConf 2017
• 59mLarge Scale MySQL Migration to PostgreSQLDimitri Fontaine · Postgres Open 2012
• 40mrails db:migrate:safelyMatt Duszynski · RailsConf 2019
• 24mMigrations under Production Load: How to Switch...Vilde Opsal · SREcon18 Europe/Middle East/Africa
• 37mDenormalizing RailsDaniel Lucraft · Ruby Manor 2: Manor Harder
• 29mScaling Rails to 2M MySQL requests per secondEugene Kenny · San Francisco Ruby Conference 2025
• 44mMongoDB to MySQL the How and the WhySarah Mei · RubyConf 2011
• 34mZero Downtime Database Migrations and DeploymentsOndrej Mirtes · Dutch PHP Conference 2018