Building production ready distributed task queue management system with celery
summary
Practical checklist for running Celery at scale: use RabbitMQ or Redis as broker depending on persistence needs; split work into pipelines and consider batching (one example picks 10 orders at once — cuts DB trips tenfold but retries affect whole batch); run IO-bound tasks with gevent/eventlet and CPU-bound with prefork; set retries (example max_retries=5) with exponential backoff, enable ack_late=True and retry_jitter; rotate workers with max_memory_per_child / max_tasks_per_child; monitor with Flower or RabbitMQ management and schedule with Celery Beat or an external scheduler.
AI-written summary · about
chapters (14)
- 0:00Introduction and speaker intro
- 1:50What are task queues
- 5:00What is Celery and why use it
- 7:28Building the system architecture
- 11:19Implementing the pipeline
- 12:02Benchmarking and performance tuning
- 14:28Optimizing I/O and CPU bound tasks
- 15:54Task distribution optimization
- 16:57Result backend and monitoring
- 17:17Adding resiliency and self-healing
- 20:53Handling memory leaks and crashes
- 22:17Monitoring with Flower
- 23:21Conclusion and key takeaways
- 24:48Q&A session
related talks
43mPainting on a Distributed Canvas: An Advanced Guide to Celery Workflows
16mTo queue or not to queue, that is the question.
17mFun, interesting and common RabbitMQ use cases
13mExplaining Distributed Systems Like I'm 5
56mImplementing distributed applications using ZeroMQ, Python and other bad guys...
32mBuilding distributed system for retail with RabbitMQ
25mAvoiding Overload in Distributed Systems
44mBuilding a Distributed Queue with Delayed Jobs on PostgreSQL