Insightful Traffic Monitoring: Harnessing Cilium for Comprehensive Network Observability
Talkat Cilium + eBPF Day North America 2024 · on Tuesday, 2024-11-12· at 09:45· Salt Palace | Level 1 | Grand Ballroom B·
summary
eBay builds a scalable traffic Observer that reads Cilium conntrack eBPF map on each node, batches reads and metric pushes, and streams enriched events via OpenTelemetry to a central metric service to assemble a service dependency graph. It handles >5,000 microservices across hundreds of clusters with clusters of 400–5,500 nodes and 50k–130k pods, meets a CPU <1 core / <1GB user-space target after tuning, reaches ~90% fidelity then near 100% by adding local enrichment reduces central load, and fixes a Canal v5.5 kernel bug encountered at extreme map sizes.
AI-written summary · about
chapters (12)
- 0:00Introduction and architecture
- 4:34Problem identification
- 6:25Service dependency graph
- 8:15Engineering challenges
- 10:03Solution design
- 17:58Improving data fidelity
- 19:43Refining architecture
- 20:14Optimizing resource usage
- 21:41Ensuring data accuracy
- 22:34Kernel bug discovery
- 23:36Lessons learned
- 24:47Q&A session
related talks
20mTackling Container Networking at Scale
34mNetwork Observability in Kubernetes with Cilium and Hubble
16mCilium Wars: Scaling Up Your Network with Jedi-like Cilium Handling Techniques - Yash Tandon
24mHigh-Scale Networking for ML Workloads With Cilium
25mThat’s Just My Cup of Tea: Configuring Cilium for Performance and Scale
16mBeyond Dashboards: Deep-Dive Network Observability with eBPF
26mDistributed Tracing: All the Warning Signs Were Out There!!!
36mCilium: Welcome, Vision and Updates