Kafka in Finance: Over 1 Billion messages a day
summary
1–2 billion market‑data ticks per day arrives from multiple vendors into a real‑time bus then into Kafka as durable replayable buffer partitioned by hashed symbol with Java producers and Python consumers. Tick collectors write into Arctic tick stores, downsample to one‑minute bars, and persist to version stores backed by MongoDB S3 and LMDB cache, totalling 100+ TB. An Arctic Native C++ rewrite claims ~2 GB/s reads and writes; retention on Kafka is tuned for days not months and vendor redundancy covers outages.
AI-written summary · about
chapters (20)
- 0:00Systematic trading at Man Group
- 1:54Data powers trading strategies
- 3:19Market data and order books
- 6:11Storing market data time series
- 7:59Research vs production environments
- 9:02Security Master reference data
- 10:26Arctic for time series storage
- 12:36Arctic Native high-performance
- 13:40Kafka streaming architecture
- 18:14Kafka tick collectors partitioning
- 22:10Kafka APIs Python processing
- 23:56Arctic APIs data retrieval
- 26:44Data ingestion pipeline summary
- 27:06Open-source at Man Group
- 28:09Hackathon community engagement
- 30:37Q&A: Kafka retention MongoDB
- 31:40Q&A: Handling data loss
- 33:36Q&A: Kafka infrastructure
- 35:22Q&A: Data reliability real-time
- 37:49Q&A: Market data granularity
related talks
04mPREVIEW: Running Kafka for Maximum Pain
33mKafka streams at scale
28mTackling Kafka, with a Small Team
35mKafka Internals I Wish I Knew Sooner
42mProcessing Millions of Events per Second with Apache Kafka on Kubernetes
31mLogging at Scale, Billion+ messages, 100K msg/sec
45mReliable Toolkit for Event-Driven Microservice Architecture
25mScaling Data Platforms: Managing Growth from Zero to Billions of Transactions