Cheating the Cloud: 50% Savings with Compression Dictionaries
summary
Internode RPC traffic uses per-message lz4 by default which often yields only ~25% reduction and can appear as plain text in captures. A sampling node trains ZSTD with external dictionaries from real RPC samples, distributes dictionaries via the cluster, and negotiates per-connection versioning so sender and receiver share dictionaries. Benchmarks show structured RPC payloads compress roughly 50% smaller on the wire versus lz4, cutting network transfer and related costs while adding modest CPU for training and deployment.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:08Customer issue with compression
- 2:54Why LZ4 was chosen initially
- 3:56Basics of compression theory
- 6:05Dictionary coding explained
- 8:16LZ77 and LZ78 algorithms
- 10:04How LZ4 works
- 11:27Zstd compression advantages
- 12:33Problem with per-message LZ4
- 13:54Dictionary-based compression solution
- 14:15Benchmark results and conclusion
related talks
24mHow Compression Will Decimate Network Costs
45mHow the Network Protocols You Choose Ultimately Affect Your Applications
15mZeta: A Scalable and Robust East-West Communication Framework in Large-Scale Clouds
08mDePIN: Pushing Decentralization Beyond Blockchain
41mEnergy Consumption and Communicating Protocols
24mHow Low Can You Go? A Comparative Dive Into gRPC and TtRPC for...
26mTuning Your Cloud: Improve Global Network Performance for Applications
18mSTYX: Exploiting SmartNIC Capability to Reduce Datacenter Memory Tax