Corruption in the Data Center! TCP can fail to keep your data safe!
summary
Datacenter packets turn into wrong payloads because intermediate switches sometimes mutate packets and recompute a valid Ethernet CRC so corrupted payloads pass wire checks, and Linux offload logic can incorrectly drop the kernel checksum validation in container setups; the incident affects many racks, takes a week or two of cleanup, and is fixed by a one-line kernel patch (submitted Feb 2016) after diagnosis with logs and rack isolation; use end-to-end CRC or TLS to prevent it.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:41The Emergency
- 2:30Insidious Corruption
- 3:44What it is supposed to happen
- 5:19Hardware related
- 7:43Bugs 1 2
- 9:06Bug 2 Explanation
- 9:28Conclusion
related talks
35mLessons Learned From Etcd the Data Inconsistency Issues
01h 12mRedundancy does not imply fault tolerance
26mDon't Let Your Kubernetes Cluster Go Wild: Ensuring Etcd Reliability
33mDeconstructing an Abstraction to Reconstruct an Outage
24mThe Butterfly Effect of a Broken Disk: Top-Down Ceph Troubleshooting To Upstream Contribution
28mWhy Are Distributed Systems So Hard?
34mEnhancing Data Integrity in Linux
23mMulti-Path Transport for RDMA in Datacenters