Painting on a Distributed Canvas: An Advanced Guide to Celery Workflows
summary
Celery canvas is dissected into concrete primitives: serializable signatures form the unit, with chains (series), groups (parallel with GroupResult), and chords (group→callback) implemented via linking and result-store coordination. Recommendation: avoid RabbitMQ as result backend (it dequeues state), prefer Redis/memcache for atomic counters or a DB otherwise; use JSON serialization not pickle. Soft/hard timeouts, acks_late and idempotent tasks reduce lost work. A custom signature called weave splits and rejoins lists to chunk Twitter’s 5,000→100-ID calls and reliably reassemble results.
AI-written summary · about
chapters (16)
- 0:00Intro
- 0:34Design Principles
- 1:54The Result Backend
- 4:43Signature
- 7:55Composition
- 11:52Group
- 14:33Cord
- 18:51Weave
- 21:49Why Weave
- 23:31Task Failures
- 25:06Rules of Thumb
- 26:48Time Limits
- 30:02Celery 311
- 31:48Questions
- 34:01Tasks
- 36:51Visualization
related talks
29mBuilding production ready distributed task queue management system with celery
16mTo queue or not to queue, that is the question.
13mExplaining Distributed Systems Like I'm 5
39mAsynchronous Processing for Fun and Profity
17mFun, interesting and common RabbitMQ use cases
01h 38mBuilding Reliable Distributed Systems
28mCRDTs and the BEAM
37mUsing Elixir for scheduling cloudnative DAG workflows