A journey of scaling high-volume low-latency stateful processing
summary
Pipeline targets 1M TPS peak and proves feasibility by hitting 3.5M TPS on the stateful dedupe operator using Apache Flink with Kafka as source after file-source checkpoint/watermark bugs force Kafka. A custom processElement sliding-1min window replaces session windows to avoid 72TB of state; pruning fields cuts per-element state from 1KB to 68 bytes (93% state reduction). File-state backend outperforms RocksDB (~30–40% faster), async external calls cost ~80–90% throughput unless made nonblocking, and checkpoints run ~3s.
AI-written summary · about
chapters (13)
- 0:00ThoughtWorks
- 0:37Problem Statement
- 1:16Architecture
- 4:55Infra considerations
- 8:30First Operator
- 8:50Deduplicator
- 10:08Dedupe Window
- 13:56Some Mathematics
- 17:59Further Optimisations
- 22:50Second Operator
- 27:17Blocked Card Check
- 31:21Topology Baseline
- 32:43Pipelines and Slots
related talks
45mApache Flink — Real-Time Data Processing
36mSources, Sinks, and Operators: A Performance Deep Dive
43mLarge Scale User Behavior Analytics by Flink
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
43mA Data Streaming Architecture with Apache Flink #bbuzz
19mBuilding Stateful Streaming Pipelines
24mEfficient, Low Latency Ingestion to Large Tables via Apache Flink and Apache Iceberg
50mStream Processing Smackdown: Kafka Streams vs. Apache Flink