15 Billion value at risk computations in 187 milliseconds
summary
Claim: 15 billion VaR computations in 187 ms. Implementation uses HBase coprocessors and direct disk reads with binary-packed position records (≈16–18 bytes) to avoid JVM heap bloat and network aggregation. Deployment runs on ~60 Amazon VMs with an Azure front end, per-worker JVMs limited to ~4 GB, CPU pegged ~95%, and HDFS short‑circuit/local reads plus chunked loads to reduce HBase metadata and disk latency. Live demo validates the latency and throughput.
AI-written summary · about
chapters (6)
- 0:00Start
- 9:40Schema Design - Positions
- 9:54Schema Design - Models
- 10:08Q1/4. Why 1 model in 1 HBase KV?
- 15:37Q4/4. Why HBase Coprocessors are used?
- 17:43How HBase helped
related talks
26mReimagine Real-time
47mReal-time Fraud Detection Challenges and Solutions
59mHadoop Essentials for the SQL Server Professional
50mFast Analytics On Big Data
10mBuilding High-Performance Financial Data Pipelines
43mCaching and Beyond - Smarter Data Processing with Java
55mApache HBase: crazy dances on the elephant back
18mBuilding a Scalable, Resilient System for Real-Time Fraud Detection