Resource Fairness: Implementing (and Deprecating) a Lease Broker
summary
Built a Temporal-based semaphore called lease broker workflow to enforce per-client concurrency for OCR activities, using signals and continue-as-new; it works but at scale signals overwhelm the broker, it becomes a costly single point of failure and underutilizes cluster capacity. Mitigations include batched signals, a dedicated broker cache, and start-rate limits, but the team ultimately deprecates the broker for compute distribution in favor of Temporal’s fair task queues (public preview, March) which raise throughput and cut cost while retaining the broker for strict concurrency/API limits.
AI-written summary · about
chapters (6)
- 0:00Introduction and speaker welcome
- 1:07Background and definitions
- 8:54Designing the lease broker solution
- 17:49Lease broker production challenges
- 23:21Deprecating the lease broker
- 28:34Q&A session
related talks
23mBodega: Localized Linearizable Reads at Anywhere Anytime via Roster Leases
28mA Deep Dive into Ray’s Scheduling Policy
26mUsing Service Brokers to Manage Data Lifecycle
25mService Brokers That Work for You!
17mTowards High-Performance Transactional Stateful Serverless Workflows with...
12mOwnership: A Distributed Futures System for Fine-Grained Tasks
47mAdopting Polyglot Flow
50mThe end of event-based architectures? Orchestration with Temporal