Beyond Zstandard: The Future of Data Compression
Talkat Open Source Summit Japan + AI_dev: Open Source GenAI & ML Summit + Automotive Linux Summit 2025 · 2025-12-08..12-10·
summary
OpenZL models compressors as a dynamically configurable graph of proven blocks and an optimizer that auto-assembles best-performing graphs in minutes, producing OpenZL serializers that embed the layout while a single universal decoder is deployed everywhere. Benchmarks show format-specific graphs match or beat specialized codecs (example: numeric SAO files see ~2x best-case ratio vs zstd level3’s ~30% and XZ’s ~60% but with XZ orders‑of‑magnitude slower). Deployed at scale with a retrain-and-ship loop, OpenZL yields doubledigit percent payload reductions at equal or higher throughput.
AI-written summary · about
chapters (16)
- 0:00Data compression evolution
- 3:30LZ77 limitations
- 5:14Why traditional algorithms fail
- 6:38Format-specific compression
- 9:28Compressing SAO star catalog
- 12:23Custom codec challenges
- 17:22OpenZ unified solution
- 21:35Adaptive compression graphs
- 22:37OpenZ data formats
- 24:41Automated compressor training
- 26:03White-box compression
- 26:47Performance vs classic algorithms
- 27:52OpenZ at Meta
- 29:39Compression as accelerator
- 30:42OpenZ roadmap
- 31:46Q&A
related talks
24mHow GZIP compression works
29mZstandard in Python 3 14 Faster Compression You Can Use Today
32mOn compression of everything in Postgres
43mzip Code: Unpacking Data Compression
01h 09mBtune: Making Compression Better
42mHoney, I Shrunk the Bits. Or how modern HTTP compression works.
18mWhy Lossless Data Compression Is Important and Fun
09mCompressing the Library of Babel!