Sleeping on the job
summary
Practical checklist for background jobs: use durable queues (Redis for fast lpush/rpop or AWS SQS via the skewer client) and monitor p50/p95/p99 with Datadog or AppSignal; deploy job class changes in blue‑red‑green two‑step deploy so old workers don’t crash on new payloads; break long work into chunks or use the iteration API / sequential perform pattern to make jobs interruptible and fair; apply concurrent execution locks (suo/Redis) and DB throttles (dbwaiter.gently) to avoid resource congestion and have an emergency blackhole key to discard runaway jobs.
AI-written summary · about
chapters (12)
- 0:00Intro
- 1:00Background jobs
- 3:00Background jobs explained
- 5:44Monitoring background jobs
- 8:44Twostep deploy
- 11:38Storage
- 16:40Sequential execution
- 20:24Job interruptibility
- 26:26Dockpiling
- 30:11Control
- 32:39Queues
- 39:07Wrap up
related talks
47mBackground Processing with
39mAsynchronous Processing for Fun and Profity
46mAsynchronous Job Processing in Java with JobRunr
38mProcrastination as a Service: Caching and Queueing
30mOn scaling Machine Learning microservices
44mMaking apps seem faster without optimizing
29mServices, Scale, Backgrounding and WTF is going on here?!???
38mImplementing Celery, Lessons Learned