The Past, Present and Future of Background Job...
summary
Panelists compare four real backends: Redis-backed Sidekiq (commercial Pro/Enterprise, batches API took years to stabilize), Solid Queue on MySQL/Postgres/SQLite (Rails default, moved away from transactional enqueue semantics), Karafka/Shoryuken using Kafka and SQS, and Postgres listen/notify plus advisory locks in GoodJob (started as 500 lines grew to ~20k for scheduling, batches, concurrency). Discussion surfaces concrete tradeoffs: after_commit enqueueing, lack of UIs and monitoring, GIL impact on multithreading, and advice to use latency-based queue names.
AI-written summary · about
chapters (13)
- 0:00Background job systems intro
- 1:07Origins and uniqueness of systems
- 4:02Why build job systems
- 8:13Original visions vs reality
- 15:41Misunderstandings about systems
- 22:51Job systems in 10 years
- 29:54Opportunities beyond jobs
- 35:38Enqueuing after transaction commits
- 41:50Origins and naming
- 46:12Patterns outside Ruby
- 49:07Ruby community support
- 50:11Active Job performance
- 54:09Closing and Q&A
related talks
28mFair by design: orchestrating background jobs in Ruby
23mLook Ma, No Background Jobs: A Peek into the Async Future
35mSidekiq at Gusto: The Parts Where People’s Jaw Drops
41mInside ActiveJob
20mSolid Queue - Default Active Job Adaptor in Rails 8
39mAsynchronous Workers to the Resque
17mActiveJob: A Service Oriented Architecture
35mProcesses and Threads - Resque vs. Sidekiq