Mixing reliability with Celery for delicious async tasks
summary
Practical rules and exact settings decide whether Celery survives production. Use on_commit so tasks enqueue only after DB commit, store large payloads in S3 not task args, enable broker confirm_publish to avoid lost publishes, and set task_acks_late true so RabbitMQ redelivers on worker death. Run a Celery Beat job every 5 minutes to reconcile DB state to queued work. Avoid long ETA/countdown tasks, prefer workflow tools (Prefect Temporal Airflow) for complex pipelines; note SQS visibility timeout defaults (30m) and Redis (1h).
AI-written summary · about
chapters (11)
- 0:00Reliable Celery tasks intro
- 1:03Why Celery reliability matters
- 3:54Avoiding task loss
- 6:00Recovering lost tasks
- 8:10Celery workflow limits
- 10:00Task loss between broker/worker
- 13:15Safe redelivery strategies
- 16:08RabbitMQ redelivery
- 18:33Worker failures handling
- 20:43Deploying Celery safely
- 22:29Celery reliability checklist
related talks
17mBuild a prod-ready distributed task queue system with celery
42mCelery on AWS ECS - art of background tasks & continuous deployment
45mPlaying tasks with Django-Celery
24mBuilding Workflows With Celery
46mA Practical Introduction to Airflow
42mHow I learned to time travel, or, data pipelining and scheduling with Airflow
25mSupercharging Pipeline Efficiency with ML Performance Prediction
36mApache Airflow in Production - Bad vs Best Practices