Challenges of Managing Real-Time Financial Market Data Storage
summary
Bloomberg migrates raw and normalized feed files from local disks to private S3-style object storage using boto3, journaling ~300 billion messages/day (~100 TB/24h) so feeds can be replayed and debugged. A local async agent chunks files (aware of 2GB memory‑mapped buffers), compresses with zlib, enforces a 30 GB disk IO pipe and 32 MB chunking limit, a local 20 MB/s limiter and a global Redis limiter of 100 MB per 5s, yielding ~40s chunk upload and meeting a two‑hour SLA.
AI-written summary · about
chapters (21)
- 0:00Real-time market data storage intro
- 7:49Market data pipeline overview
- 9:39Why market data files matter
- 12:07Backup system challenges
- 14:39Market data volume growth
- 16:44Global scale & bandwidth limits
- 18:10Why keep files for market data
- 20:20Scalable storage requirements
- 21:27Cloud storage architecture
- 23:38Network zones & security
- 25:08Feed handler agent design
- 27:02Agent file processing & async I/O
- 27:43Rate limiting disk & network
- 29:30Bandwidth optimization
- 32:05Memory-mapped files in agents
- 34:12Market session handling
- 36:00Client data access interfaces
- 37:52Monitoring and service level agreements
- 38:42Decouple storage from feed handlers
- 39:24Future scalability & performance
- 40:49Summary
related talks
28mDataEngBytes 2023 - PER-T1-04 - Nathan Glover
21mOptimize and manage data on Amazon S3
52mMaking sense of Crypto Market Data in real time
26mArchitecting real-time transactional data lake
08mReal-Time Data Systems: Speed, Scale, and Resilience
19mAdoption of Big Data in Securities Processing Industry A Developer's Perspective
48mHyper Speed! When Big Data Blooms
31mScaling Autonomous-Driving R&D: A 100 PB, 100 Billion-File Open-Source Elastic File...