~/talk

Celery and Kubernetes for a Fast, Scalable, and Robust Workflow Orchestration

Talkat PyCon India 2025 · 2025-09-12..09-15· [python, kubernetes]
linkedin:param-rajani
slides:none on file

Describes a production migration from AWS Lambda and Step Functions to Celery Canvas with Kubernetes that replaces state transitions and per-invocation costs, runs specialized Celery workers as separate pods, and uses pre-fork for CPU tasks and gevent for IO tasks. Observability uses Sentry for errors and an EFK logging pipeline. Redis proves more stable than SQS in their runs. End-to-end latency drops from 30s → 8s, showing the setup works at scale with higher operational overhead and DevOps investment. Redis broker stable

AI-written summary · about

  1. 0:00Introduction and speaker background
  2. 1:25Workflow orchestration challenges
  3. 3:12Celery as an open-source alternative
  4. 4:37What is Celery and task queues
  5. 6:28Celery workers and task execution
  6. 7:31Celery Canvas for workflows
  7. 8:15Celery vs AWS Step Functions
  8. 11:58Compute and scaling with Kubernetes
  9. 12:43Production architecture overview
  10. 14:09Worker types and resource allocation
  11. 16:40Celery deployment on Kubernetes
  12. 18:24Observability in Celery workflows
  13. 19:30Error tracking with Sentry
  14. 20:11EFK stack for logging
  15. 21:41Performance and scalability results
  16. 22:45Cost and control comparison
  17. 24:54When to choose Celery vs Lambda
  18. 25:40Q&A session
~/search
type to search 6,518 conferences, 143,918 speakers and 192,133 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
← →
on a talk: 5 seconds back / forward
0–9
on a talk: jump to 0%–90% of the recording
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.