Lessons From Our First Trillion Messages with Flow
summary
Describes running Flow (on GenStage) to process a trillion messages from SQS with backpressure, visibility-window acks and batched deletes limited to SQS batches of 10. Keeps a single message struct through the pipeline, isolates side effects to the end, and uses partition+reduce to ack every 10 messages or 5s. Tunes max_demand, stages (often up to 100 stages for IO-heavy mappers), measures with instrumentation/new_plot, and prototypes a near-drop-in Broadway port.
AI-written summary · about
chapters (44)
- 0:00Intro
- 1:32Adoption story
- 2:10General flow
- 2:57hiccups
- 3:16What Flow do
- 3:44Abstraction
- 3:59Code Example
- 6:55Slack
- 7:18Five humans
- 8:20Organize your flow
- 9:20Know your code
- 10:11Code psych
- 10:27SQS
- 11:21Flow
- 12:03Flow Enum
- 13:17Partitioning
- 14:06Voice in your head
- 14:33Use the token
- 14:50Message struct
- 15:23Code
- 16:20Message Types
- 17:35Errors
- 18:34Partition Reduction
- 19:17Isolate Side Effects
- 19:49Sagas
- 21:33Measuring
- 22:36Tweaking
- 22:50Main Flow
- 23:03Producer Consumer
- 24:17Mappers
- 24:51Partition reducer
- 25:29Stages
- 27:00SQS Producer
- 27:27Dependencies
- 28:17Review
- 28:43Other Lessons
- 29:21Broadway
- 30:22Broadway Performance
- 33:00Supervision Trees
- 34:05What does this mean
- 34:40Flow still applies
- 34:50Defining stages
- 36:48Thank you
- 37:18Change the Dog
related talks
22mHigh throughput data processing with GenStage
45mKEYNOTE: Announcing Broadway
31mLogging at Scale, Billion+ messages, 100K msg/sec
34mA journey of scaling high-volume low-latency stateful processing
44mData Processing With Microservices
48mTaming & Composing High Performance Stream-Oriented Processing Systems
31mFlink @WORKDAY: scaling of ingestion data pipelines to S3
32mStream Processing at Lyft with Flink and Beam