Celery and Kubernetes for a Fast, Scalable, and Robust Workflow Orchestration
summary
Describes a production migration from AWS Lambda and Step Functions to Celery Canvas with Kubernetes that replaces state transitions and per-invocation costs, runs specialized Celery workers as separate pods, and uses pre-fork for CPU tasks and gevent for IO tasks. Observability uses Sentry for errors and an EFK logging pipeline. Redis proves more stable than SQS in their runs. End-to-end latency drops from 30s → 8s, showing the setup works at scale with higher operational overhead and DevOps investment. Redis broker stable
AI-written summary · about
chapters (18)
- 0:00Introduction and speaker background
- 1:25Workflow orchestration challenges
- 3:12Celery as an open-source alternative
- 4:37What is Celery and task queues
- 6:28Celery workers and task execution
- 7:31Celery Canvas for workflows
- 8:15Celery vs AWS Step Functions
- 11:58Compute and scaling with Kubernetes
- 12:43Production architecture overview
- 14:09Worker types and resource allocation
- 16:40Celery deployment on Kubernetes
- 18:24Observability in Celery workflows
- 19:30Error tracking with Sentry
- 20:11EFK stack for logging
- 21:41Performance and scalability results
- 22:45Cost and control comparison
- 24:54When to choose Celery vs Lambda
- 25:40Q&A session
