DIY Async Message Pump: Lessons from the trenches
summary
Built an async .NET message pump that hooks to MSMQ, RabbitMQ, Azure Service Bus, SQS, Kafka and SQL Server and surfaces concrete failures: unbounded concurrency kills DB pools (demo shows tens of thousands of tasks in 100ms), a 4,000ms handler inflates queue wait time, and a SemaphoreSlim plus slot throttling (think SemaphoreSlim toll-lane) reclaims stability. Covers DTC vs receive-only semantics, visibility timeout mechanics, send-via batching to cut cloud operation costs, and practical tradeoffs for exactly‑once, idempotency and shutdown.
AI-written summary · about
chapters (11)
- 0:00Introduction and project background
- 4:39Architecture challenges and solution
- 13:52First message pump implementation
- 18:08Concurrency and unbounded risks
- 22:00Adding concurrency control
- 27:15Evolving the message pump
- 30:49Transaction management deep dive
- 40:44Cloud transaction semantics
- 43:15Cost and latency trade-offs
- 47:11Distributed system complexities
- 50:01Lessons and takeaways
related talks
56mDistributed Messaging patterns
02h 42m!!! Do not publish or share !!!!
48mReliable messaging in the world of actors
40mEverything Will Flow
40mUsing ZMQ for Fun and Profit
30mDo Distributed Systems, they said. It'll be fun, they said
57mReliability Nirvana
22mAsynchronous Computing at Scale: Overview and Learning