Understanding Probabilistic Data Structures with 112,092 UFO Sightings
summary
112,092 UFO reports from a public CSV (screen-scraped by Tim Renner) become a testbed for concrete probabilistic tools: Bloom filter for membership with a demoed 0.001 false-positive target, sizing formulas and hash-count tradeoffs; MinHash using trigram shingling and three hash functions to estimate Jaccard similarity; and Top‑K (heavy keeper + min‑heap) with decay (example B≈1.05) to maintain trending shapes/words. A Python/Flask pipeline loads the file into Redis (RedisBloom) and serves a working web endpoint showing top shapes and words.
AI-written summary · about
chapters (12)
- 0:00Welcome
- 1:22Why probabilistic structures matter
- 7:26Deterministic vs probabilistic
- 15:10Accuracy trade-offs
- 21:13Key probabilistic structures
- 24:25Bloom filters use cases
- 31:10Bloom filter collisions
- 33:56Bloom filters in Redis demo
- 37:12MinHash for similarity
- 43:34Top K and Heavy Keeper
- 52:03Top K with UFO sightings demo
- 56:36Resources and closing
related talks
53mUnderstanding Probabilistic Data Structures with 112,092 UFO Sightings
55mUnderstanding Probabilistic Data Structures with 112,092 UFO Sightings
52mUnderstanding Probabilistic Data Structures with 112,092 UFO Sightings
26mNo, Maybe and Close Enough: Using Probabilistic Data Structures in Python
26mNo, Maybe and Close Enough! Probabilistic Data Structures with Python and Redis
01h 00mHyper Speed: When Big Data Blooms
37mWhen Probably is Good Enough
51mProbabilistic Data Structures with Redis Bloom