There’s A Bug in My Service Mesh! What Do You Do When the Tool You Rely On...
summary
Paybase runs ~50 gRPC microservices on Kubernetes and deploys a Linkerd proxy sidecar that causes 502s by mutating headers; debugging reveals two root causes: a panic on HPACK continuation frames with repeated headers and looking up an evicted header index. Repro uses linkerd tap and tcpdump plus increased proxy logging, a debug sidecar, tshark and CPU/top checks. Fixes land in Linkerd (bugs in Rust h2 handling) and diagnostics improve; collaboration with maintainers speeds resolution.
AI-written summary · about
chapters (29)
- 0:00Introduction
- 0:35Ana Calin
- 0:47Risha Mars
- 0:59Agenda
- 1:33Mushrooms
- 2:20Open Source Projects
- 2:42Communication Matters
- 3:35What is a Service Mesh
- 4:16Control Plane
- 5:02Paybase
- 7:17Example
- 8:07Troubleshooting Framework
- 8:51Reliability Engineering Book
- 9:51Decision Tree
- 13:13System Diagram
- 16:39Filing an Issue
- 17:40Triaging
- 17:59Reproducibility
- 18:22Impact
- 18:43Where is the bug
- 19:16Anas bug report
- 19:48TCP dump
- 20:35The diagram
- 21:05How multiplexing works
- 22:00Header compression
- 22:56Bugs
- 23:30Diagnostic Tools
- 24:22Troubleshooting Tools
- 27:38Working with Android
related talks
20mTop new CNCF projects to look out for
23mScala the Cloud Native Way...
31mWhy the future of the cloud will be built on Rust
29mKubernetes Failure Stories and How to Crash Your Clusters
26mTop new CNCF projects to look out for
09mLightning Talk: Hacking Kubernetes Networking to Create Telco Solutions
44mStretching the Service Mesh Beyond the Clouds
10mOpening Remarks