Data Processing for Market Regulation
summary
FINRA aggregates trade feeds from 18 national exchanges and normalizes equities options and most fixed income data (excluding U S Treasuries) to run rule-based anomaly detection for wash sales insider trading and other violations. The pipeline processes about 30 billion records per day with peaks around 55 billion so the infrastructure is migrating to the cloud for elastic compute and exploring SQL-on-Hadoop and engines like Presto Impala Spark to speed analysis and investigator navigation through billions of rows.
AI-written summary · about
related talks
19mAdoption of Big Data in Securities Processing Industry A Developer's Perspective
33mReal-time Trade Data Processing and Machine Learning at Giga-Sca...
24mRetail Transactions at scale for AI - From PoC to production
42mChallenges of Managing Real-Time Financial Market Data Storage
08mReal-Time Data Systems: Speed, Scale, and Resilience
46mNoSQL, Data Processing, and Business Transparency with RavenDB
53mData Mesh in Practice - Assuring Data Quality at Scale
28mApache NiFi, Kafka, RisingWave, Iceberg + Stock Data