A story about debugging of one Go application

on YouTube ↗35m

" summary

Debugging a Go CSI driver for Kubernetes uses concrete observability: initially add gRPC trace interceptors to see per-request spans, then switch to metrics when traces lack history, instrument with Prometheus+Grafana and add many domain metrics; centralize JSON logs into the ELK stack and enable a runtime log level toggle via SIGUSR1. Metrics point to a slow “publish volume to node” handler, logs show repeated mount errors "no such file or directory" caused by a cleanup race and retries, and increasing timeouts fixes the regression.

AI-written summary · about

" chapters (12)

1. 0:00Introduction
2. 3:08What is CSI driver
3. 6:39CSI driver development challenges
4. 10:11Performance degradation problem
5. 15:46Tracing with OpenTelemetry
6. 17:32Metrics with Prometheus
7. 20:44Dynamic log management
8. 22:53Centralized log storage
9. 25:27Solution and results
10. 27:55Alternative debugging tools
11. 28:16Remote debugging in Go
12. 31:53Q&A

" related talks

• 01h 00mDebug like a pro on KubernetesGianluca Arbezzano · ScaleConf 2019
• 40mDebugging in Golang Apps Inside K8s EnvironmentAndrii Soldatenko · KCD Austria 2023
• 31mDebugging Go: from zero to Kubernetes - Michele CaciMichele Caci · ContainerDays 2023
• 30mStreamlined Troubleshooting in KubernetesLena Fuhrimann · Kubernetes Community Days Zürich 2023
• 01h 32mTutorial: Debug Your Kubernetes AppsArun Gupta, Re Alvarez Parmar · KubeCon + CloudNativeCon North America 2019
• 24mDebugging at Scale in Production - Deep into your ContainersShai Almog · Conf42 Cloud Native 2022
• 50mTroubleshooting & Debugging Microservices in KubernetesRay Tsang, Robert Kubis · GOTO Chicago 2019
• 29mWhen Kubernetes Misbehaves – Pitfalls and Lessons LearnedItiel Shwartz · ContainerDays 2022