Insights from Engineering Sketches for Production and Using Sketches at Scale
summary
Apache DataSketches originates at Yahoo in 2012, now shipping ~15 sketch types across Java, C++ and Python and producing a billion sketches per day in production. Key claims: sketches must be mergeable across K with no extra error, start tiny and grow sublinearly, and give accurate estimates down to cardinality 1 for power‑law groupings. Implementations combine multiple estimators (HIP, Poisson, polynomial smoothing), offer HLL 4/6/8 and KLL/CPC/Theta families, provide confidence intervals, and trade update/merge/serialization costs for space and accuracy.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 1:06Engineering sketches matter
- 3:40Decade of sketch production
- 7:30Power-law data storage
- 12:08Growth strategies for sketches
- 16:35Error behavior in sketches
- 22:25Fixing HLL error spikes
- 33:25Confidence intervals and guarantees
- 39:38HLL vs newer sketches
- 41:51Tuple sketches for analysis
- 44:30Merging sketches
- 46:19Production sketch library
- 47:51Q&A
- 51:25Theory vs practice
- 57:24Graph sketching challenges
- 59:54Memory vs accuracy trade-offs
- 1:01:53Closing remarks
related talks
29mSketching in the Browser
19mSketchExpress: Remixing Animations For More Effective Crowd-Powered Prototyping Of Interactive ...
24mSketching in the Browser
49mFrom Doodles to Dashboards, Data Flows, and Docs
21mSemi-Automated SVG Programming via Direct Manipulation
30mData Skills for Data People
29mEffective design and engineering collaboration
02h 52mLive wireframing: lean BI using AI