There's a Bug in My Service Mesh! What Do You Do When the Mesh is At Fault?
Talkat KubeCon + CloudNativeCon North America 2019 · on Wednesday, 2019-11-20· at 14:25· Hall D - San Diego Convention Center·
summary
Paybase runs 50+ microservices and adds Linkerd sidecar proxies for gRPC load balancing; deployment surfaces obscure bugs where headers get split across HTTP/2 frames causing a panic and another bug where evicted header table entries use the wrong index. Diagnosis uses Linkerd proxy logs at debug level, tcpdump and linkerd tap to inspect frames, and a debug sidecar with tshark. Bugs live in the h2 Rust library, are fixed upstream, and fixes land after reproducible test cases and detailed bug reports with logs and version matrices.
AI-written summary · about
chapters (16)
- 0:00Intro
- 1:56Nested Mushroom
- 3:06Communication
- 4:24What is a Service Mesh
- 5:17LinkerD
- 5:42PayRaise
- 6:24Kubernetes
- 9:04Brian Redman
- 9:48Google Azhar
- 11:07What to do if you have an error
- 14:10Understanding your system
- 16:55File a bug report
- 18:43How to diagnose a bug
- 20:44H2 background
- 22:30Anas original blog
- 25:59Summary
related talks
30mMuseum of Weird Bugs: Our Favorites From 8 Years of Service Mesh Debugging
30mService Mesh Is Still Hard. The Things we all Did Wrong & Tales of Woe
31mClean-up Your Service Mess with Service Mesh
31mThe Future of Service Meshes: Sidecarless or eBPF?
04mInterview
39mDo you need a Service Mesh?
29mYou Probably DON’T Need a Service Mesh!
44mService Mesh - The New Single Point of Failure