Hyper Speed: When Big Data Blooms
summary
Processes about 65,000 JSON CSP/telemetry reports per second and uses a Bloom filter in Redis to answer “seen before?” queries in memory, returning definite no or probable yes with a controlled 1% false positive rate. Design targets ~7 billion distinct entries so the bit array is ~8.4 GB; k=7 hash functions. Also runs count‑min sketches, top‑K and HyperLogLog for frequency and cardinality, deployed and currently testing at production scale.
AI-written summary · about
chapters (25)
- 0:00Intro
- 0:43Title
- 1:14The Problem
- 2:27Crosssite Scripting
- 3:34Coinhive
- 4:33Magecart
- 5:57Content Security Policy
- 6:27JavaScript
- 7:49Content Security
- 9:55Report Uri
- 12:09CSP Reports
- 17:32How can we do this
- 18:24Bloom Filters
- 19:56Inserting Items
- 24:33Probabilities
- 30:31Bloom Filter Width
- 32:19Bloom Filters Online
- 35:16Akamai Case Study
- 37:37Google Safe Browsing
- 40:57Medium
- 43:43Counting
- 45:00Sketch
- 47:22TopK
- 49:21Hyperlog Log
- 51:29QA Session
related talks
48mHyper Speed! When Big Data Blooms
01h 00mHyper Speed: When Big Data Blooms
49mHyper Speed! When Big Data Blooms
37mWhen Probably is Good Enough
25mHigh on Cardinality, Low on Memory
26mNo, Maybe and Close Enough! Probabilistic Data Structures with Python and Redis
10mDon’t forget to sketch! Running with large datasets
09mBig Data, Big Performance