Mo' Jobs Mo' Problems: Lessons Learned Scaling to Millions of Jobs an Hour
summary
Builds a concrete story: moves from syslog loss to replacing pipelines with RabbitMQ for durable event ingest and SQS for cheap highly available queues, implements a custom processor called chore, and proves it handles over a million jobs an hour. Explains specific failures encountered — Ruby system stack errors, DNS C extensions blocking the GIL, fork mutex races, pipe IPC bottlenecks — and the exact tradeoffs and concurrency choices that finally scale and stabilize the system.
AI-written summary · about
chapters (39)
- 0:00Intro
- 1:04What is Tapjoy
- 2:20Background
- 2:38Everything was going well
- 2:49Good news
- 3:59Our fault
- 4:50Event processing
- 5:56This is not awesome
- 6:07Change is hard
- 6:25Message queues
- 6:54Cues
- 7:55Durability
- 8:23Availability
- 9:01Cost
- 9:34Three choices
- 10:25Breaking down Rabbit
- 11:03Breaking down SQS
- 11:48Breaking down Redis
- 12:50Two use cases
- 13:54Ruby Conference
- 14:48What is a Job Processor
- 15:13What is Scale
- 15:44Ruby Concurrency
- 16:37Rabbit Problems
- 17:38Fibers
- 22:20What we learned
- 23:10What it takes to process jobs
- 23:50SQS
- 24:48Concurrency
- 25:00Threads
- 26:39Forks
- 27:14Using Threads
- 29:01Problems
- 34:38Bugs
- 35:30Lessons Learned
- 37:20Job Queue Library
- 38:43Job Queue Issues
- 40:45Job Processing Lessons
- 42:58Outro
related talks
31mLessons Learned Running Sidekiq at Scale
11mRunning Jobs at Scale
01h 03mRuby at Scale
27mScaling Ruby Applications with Mesos and Docker
24mPlan to scale or plan to fail: an evidence-based approach for improving systems performance
30mACIDic Jobs: Scaling a resilient jobs layer
34mHow to scale a Ruby webservice
18mRuby Messaging Patterns