Fair by design: orchestrating background jobs in Ruby
summary
Shows that Rails background processors (Sidekiq, GoodJob, SolidQueue) cannot natively enforce multi-tenant fairness at scale and demonstrates four concrete fixes: shuffle shards to reduce collision probability, interruptible iteration to slice large jobs and requeue progress, leaky bucket throttling to demote greedy tenants into slower queues, and a working manager that uses virtual per-tenant queues (a central scheduler/ledger) which achieves near-perfect fairness at the cost of added scheduler throughput.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 1:52Unfair job queues problem
- 5:07Fairness in job processing
- 7:37Limitations of processors
- 11:43Four fairness strategies
- 12:04Shuffle sharding
- 15:19Interruptible iteration
- 17:29Leaky bucket throttling
- 20:23Centralized scheduler
- 23:56Concurrency vs fairness
- 25:19Choosing fairness strategy
- 26:45Resources and closing
related talks
55mThe Past, Present and Future of Background Job...
39mAsynchronous Workers to the Resque
27mFrom Resque to SolidQueue - Rethinking our background jobs for...
32mSimulation Testing for Background Jobs
23mLook Ma, No Background Jobs: A Peek into the Async Future
41mInside ActiveJob
35mSidekiq at Gusto: The Parts Where People’s Jaw Drops
34mShared Nothing Architecture - How to Scale to Infinity