Speeding up LZ4 decompression
summary
The talk examines methods to speed up LZ4 decompression. It covers low-level CPU-specific optimizations ("black-magic") and data-science-driven techniques ("white-magic") to improve decompression performance across a wide range of CPU models.
AI-written summary · about
chapters (16)
- 0:00Introduction to LZ4
- 1:10Issue with fastLZ4 function
- 7:18Why data compression is needed
- 7:39How LZ4 works
- 9:47Optimizing data copying
- 11:13memcpy problems in LZ4
- 14:46Optimizing via memcpy
- 17:42Complex copy cases
- 19:28Optimizing via SIMD
- 21:38x86 shuffle instruction
- 24:51Removing boundary checks
- 26:39Testing optimizations
- 30:37CPU dependency
- 32:48Multi-armed bandit for algorithm selection
- 37:30Test results
- 39:43Conclusion
related talks
46mUnconventional approaches to performance in Ozon's search engine
01h 14mDistributed and concurrent optimization for machine learning
16mAccelerating Design Space Exploration for LLM Training Systems with Multi-experiment...
41mMaximizing Mutual Information in Flash Memories
38mTechniques and Tools for a Coherent Discussion about Performance in Complex...
31mSome Novel Computational Models to Think About
45mFaster or better designed? Choose any two!
27mThe Four Pillars of Large-Scale Decentralized Pre-Training: Methods, Bottlenecks...