What does "high priority" mean? The secret to happy queues
summary
Argues that queues must be organized by explicit latency guarantees (e.g., within 1 minute, within 10 minutes) rather than vague priority labels; implements contracts where queue names set max start latency, alerts fire on breaches, and autoscaling triggers when latency hits ~50% of the allowed window. Sets per-queue runtime caps (rule of thumb: 1/10th of latency target), creates a lint rule to enforce placement, migrates ~60 queues down to 9, and reports fewer late jobs and much lower infra cost.
AI-written summary · about
chapters (10)
- 0:00Meet Sally: The Engineer with a Problem
- 0:40A Radical Solution and Some History
- 1:48The Queue Dilemma Begins
- 2:30Joey's Attempts to Fix the Queues
- 4:13Marianne's Insight: Separate Queues for Different Purposes
- 5:41The Growing Pains of a Scaling Company
- 8:36The Real Problem: Latency
- 13:05Sally's Brilliant Idea: Organizing Queues by Latency
- 22:00Implementing the New Queue Structure
- 26:04Q&A: Applying Queue Principles to Online Jobs
related talks
33mMind Your P’s and Queues
40mEverything Will Flow
40mSystem Performance and Queuing Theory - Concepts and Application
25mAvoiding Overload in Distributed Systems
32mTaking Care of Your Control Plane With API Priority and Fairness and Resource Quotas
39mUnderstanding minimal viable quality promotes craft
20mBetter Resilience Through Self Tuning
42mHow Flow Works and other curiosities