High-Performance Market Data in a Cloud-Native World
Talkat Open Source in Finance Forum New York 2022 · on Thursday, 2022-12-08· at 13:55· HUB 1, New York, NY, USA·
summary
Concrete claim: a schema-driven transcoder called market-data-transcoder (described as ffmpeg for market data) reads binary feeds like SBE/ITCH/FIX, preserves every field with lossless transcoding, handles real-world quirks (ITCH nanosecond timestamps via message handlers), and auto-creates matching sinks (BigQuery tables, Avro files, Pub/Sub) so raw depth-of-book and full-session packet captures land queryable in the cloud. Works with Kafka/PubSub for ordering, reduces manual ETL, and enables global distribution and analytics without data loss.
AI-written summary · about
chapters (11)
- 0:00Scaling market data in cloud
- 2:12Wall Street vs Silicon Valley paradigms
- 5:59Finance data models to cloud-native
- 8:47Schema transport and encoding
- 12:39Life of a transcoded message
- 15:26Infrastructure as schema
- 18:15Move financial data to cloud
- 20:01Market data transcoder tool
- 23:13Protocol edge cases in transcoding
- 27:06Key concepts and tool recap
- 30:17Q&A on conflation and analytics
related talks
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
36mThe MODERN Modern Data Stack: Building an Open Distributed Data Warehouse Beyond...
20mBuilding realtime efficient queries in an IOT platform
08mReal-Time Data Systems: Speed, Scale, and Resilience
29mThe need for speed: Data streaming architectures
39mSuper Fast Real-time Data Processing on Cloud-Native Architecture
26mArchitecting real-time transactional data lake
25mMixing data streams at scale