Reliability in Finance: Delivering Quality Market Data at Scale
summary
Describes a production pipeline that adds checkpointed visibility, automated milestones and lineage so files are validated on arrival, processed through RabbitMQ/CFKA and monitored with Prometheus, and anomalous trends are caught with CUSUM anomaly detection. Gives concrete mitigations: pause runs until required files appear, global per-customer rate limits, an emergency off switch and a 20% demo tenants safety valve in production to buy headroom. Cites a stale feed causing bad prices and a $440M rapid-loss maintenance incident to justify SLO-driven runbooks.
AI-written summary · about
chapters (15)
- 0:00Bloomberg and market data challenges
- 3:35Financial data failures lessons
- 5:00SRE for reliable data pipelines
- 7:29Visibility and transparency in delivery
- 11:52Checkpoints in data pipelines
- 16:08Deployment disasters guardrails
- 17:55Emergency runbooks and response
- 18:37Data validation and lineage
- 20:43Scaling reliability metrics
- 23:13Anomaly detection and ML
- 25:19Safety valves and load management
- 27:06Sales collaboration for capacity
- 28:10Rate limiting for fair access
- 29:55Key lessons for financial systems
- 31:43Closing remarks and hiring
related talks
56mGetting all the 99.99(9) you always wanted
45mResilient Predictive Data Pipelines
58mCultivating Production Excellence
27mContinuous Reliability. How?
19mObservability for Data Pipelines: Monitoring, Alerting & Tracing Lineage
43mGetting Your System into Production and keeping It There
16mPredictive Observability: Prevent Supply Chain Outages
52mPerformance & Stability Anti patterns