Surviving Black Friday: 329 billion requests with Falcon!
summary
Falcon replaces Unicorn for Shopify storefront rendering and carries 329 billion requests over Black Friday weekend, peaking at 1.42M req/s, with a fiber-based Async worker model, a long tasks semaphore to limit CPU contention, and a supervisor that monitors memory and utilization. Failures include RD Kafka C-extension segfaults, Spanner GRPC deadlocks, and a notification-pipe “silent death”; fixes are gem and buffer caps, kill-as-last-resort for stuck GRPC, and atomic writes plus timeouts. A per-pod Outbox per-pod Kafka proxy consolidates thousands of connections into one, enabling rollout and stable production performance.
AI-written summary · about
chapters (22)
- 0:00Introduction to Falcon
- 0:35The bet we made
- 1:58Traffic and scale challenges
- 2:20Why Unicorn wasn't enough
- 3:23How Falcon works
- 6:59Controlling concurrency with long tasks
- 8:42Fiber-safety in Ruby code
- 9:27Phased rollout plan
- 11:11Scale tests and early results
- 12:33The darkest week of the project
- 13:15Crisis one: Kafka side faults
- 14:20Crisis two: Spanner deadlocks
- 15:46Crisis three: Silent death
- 18:36Rollback and deadline pressure
- 19:18Kafka cardinality issues
- 21:02Building Outbox solution
- 22:05Go/no-go meeting
- 22:47Final scale tests
- 23:29BFCM weekend begins
- 23:49Peak traffic and performance
- 25:36Five key lessons learned
- 26:59What's next for Falcon
related talks
25mPuma or Falcon What's the Best Choice for Scalability?
25mUnderstanding Ruby Web Server Internals: Puma, Falcon, and...
33mAsynchronous Rails
21mConquering Massive Traffic Spikes in Ruby Applications with Pitchfork
37mRhebok, High Performance Rack Handler
27mIn-Depth Ruby Concurrency: Navigating the Ruby concurrency landscape
36mHigh Availability at Braintree
23mLook Ma, No Background Jobs: A Peek into the Async Future