DIY Async Message Pump: Lessons from the trenches
summary
Built a message pump that started as a single-loop async receiver, then exploded into unbounded concurrency (tens of thousands of concurrent tasks on an i7) and overwhelmed resources; fix is a SemaphoreSlim concurrency gate tuned to the weakest link (DB connection pool default ~100). Covers transports: MSMQ+SQL with DTC, RabbitMQ and receive-only transactions (partial side effects), Azure Service Bus atomic sends, SQS/storage queues using 30s visibility timeouts, and cloud limits (256 KB batches, 13M free ops/month) — batching and precise transaction placement are essential.
AI-written summary · about
chapters (15)
- 0:00Introduction to async pumps
- 1:06Patient record system challenges
- 4:56Build vs buy messaging library
- 7:23Basic message pump code
- 11:13Handling rush hour traffic
- 13:20Adding concurrency to pump
- 15:25Unbounded concurrency pitfalls
- 18:53Limiting concurrency with semaphores
- 22:59Extending the message pump
- 27:09Transaction handling in pumps
- 33:07Moving to cloud transports
- 37:41Cloud latency and cost savings
- 40:28Batching messages efficiently
- 43:14Advanced pump requirements
- 45:39Monitoring distributed systems
related talks
01h 00mDIY Async Message Pump: Lessons from the trenches
56mDistributed Messaging patterns
01h 01mBeyond Pub/Sub - Advanced Messaging Patterns
01h 01mThe Unsung Hero of Modern Software: Asynchronous Messaging
01h 03mBeyond Pub/Sub - Advanced Messaging Patterns
47mEverything Loosely Coupled - Advanced Integration Patterns and Their Trade-Offs
01h 02mMessaging Patterns for Modern Software Solutions
38mMessaging Patterns to Transform Your Cloud Architecture