5 Steps to Resilient Job Queues
summary
Five concrete ops rules: encode latency SLOs into queue names (e.g. 5s for MFA, 5m for thumbnails, 5h for legal mail), monitor the single metric that matters—queue latency—and wire latency-based auto scaling so workers scale up when SLOs are breached and back down when idle. Isolate outliers with dedicated high-memory workers and let those worker pools scale to zero when unused. The claim: these five steps cut backpressure and idle spend and keep queues flowing.
AI-written summary · about
related talks
31mLessons Learned Running Sidekiq at Scale
27mPrioritization justice: lessons from making background jobs fair at scale
11mRunning Jobs at Scale
21mHarmonizing Efficiency and Practicability: Optimizing Resource Utilization in...
19mAWS LAMBDA: FAAS TRICKS TO MANAGE AUTOSCALING GROUPS
44mMo' Jobs Mo' Problems: Lessons Learned Scaling to Millions of Jobs an Hour
30mBuilding Scalable & Resilient Background Jobs for FinTech
30mACIDic Jobs: Scaling a resilient jobs layer