Streaming financial market data through WebSockets
summary
XM’s backend streams market prices via AWS WebSocket API Gateway, Kafka and Kubernetes, using Java 21 with ZGC and aggressive batching; a DynamoDB subscription table is replaced by a sharded Kinesis stream plus statefulset consumers to eliminate hot-partition throttling. Payloads shrink so a 32KB post carries ~80 prices, async HTTP clients have zero retries, and adaptive partition assignment evens load. System sustains 120,000 prices per second and meets P99 < 600ms, scaling to 30,000 concurrent connections.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 1:30Who XM is
- 3:15Live demo trading platform
- 4:39System requirements SLAs
- 5:21Streaming financial data challenges
- 7:49XM real-time event architecture
- 9:13WebSocket API Gateway AWS
- 12:00Data flow client to price
- 18:19Java service architecture limits
- 20:05Scaling Kafka and Kubernetes
- 21:34Hot partitions impact
- 23:00Kubernetes optimizations
- 26:57DynamoDB bottlenecks fixes
- 29:47DynamoDB to Kinesis switch
- 33:16Kinesis shard scaling stateful
- 35:45Other solutions limits
- 36:47Fixing Kafka partition data
- 37:29Performance testing results
- 38:32Key scaling lessons
- 39:56Performance testing best practices
related talks
31mStreaming: Real-Time Architecture for Connected Products in Automotive
29mAPIs and Event Streaming Dualty
23mDeliver real-time data to customer using Streaming APIs
01h 28mIntroduction to WebSockets
16mBuilding scalable IoT systems with MQTT and AWS
23mCustomer Journey with streaming data on AWS
42mReal-time Energy Trading: Alpiq's Strategic Data Platform on AWS
57mWhy and how to transform a REST API into a Data Streaming API