Running Jobs at Scale
summary
Shopify faces jobs that run seconds for small data but take minutes to weeks for large merchants, and frequent deploys every 20 minutes break long work. The fix is concrete: split a job into a collection and a per-record unit, persist a cursor between interruptions, and resume; works with ActiveRecord relations and CSV enumerators. Results: progress tracking, safe parallelism, load throttling, tolerance for short-lived cloud instances, and successful retries without losing progress.
AI-written summary · about
chapters (4)
related talks
40mHow Shopify Scales Rails
17mHow We Deploy Shopify
30mBuilding Scalable & Resilient Background Jobs for FinTech
32mZero Downtime Deploys Made Easy
31mLessons Learned Running Sidekiq at Scale
21mTesting Rails at Scale
34mIndustrial Rails
44mMo' Jobs Mo' Problems: Lessons Learned Scaling to Millions of Jobs an Hour