Network Flow Aggregation: Pay for the Logs You Care About!
summary
Agent-side flow aggregation in Cilium 1.19 serializes selected fields into a deterministic key, collapses per-direction flows into counts (ingress/egress/unknown), and exports aggregated records on a configurable aggregation interval. Tests show modest CPU, slight memory increase, improved disk tail latency, and large storage savings: synthetic runs reach ~87% size reduction while production samples average ~45% reduction; a live demo compresses 1.7GB down to 0.4GB with ~62% fewer rows. Configuration uses a field_aggregate list to preserve identity and semantics.
AI-written summary · about
chapters (15)
- 0:00Network flow aggregation intro
- 1:45Observability costs and logs
- 3:12Log filtering limits
- 4:17Psyllium 19 flow aggregation
- 5:45Aggregation architecture
- 7:10Why logs keep meaning
- 8:16Aggregation performance impact
- 10:03Compression results analysis
- 11:27Production aggregation effectiveness
- 12:08Balancing aggregation fields
- 13:55Fine-tuning for production
- 15:01Key takeaways and future
- 16:48Demo: aggregation config
- 20:44Demo: log volume reduction
- 21:50Visualizing flows in Grafana
related talks
16mNetwork Flow Data in the Cloud
21mTransparent and Flexible Network Management for Big Data Processing in the Cloud
20mAggregating Metrics In-Flight: Challenges, Opportunities
41mNetwork Observability
38mAggregating Metrics In-Flight: Challenges and Opportunities
21mYour Coflow has Many Flows: Sampling them for Fun and Speed
19mA History of Automatic Aggregations
35mNetwork Flow Monitoring in Kubernetes with Contiv/VPP CNI and Elastic Stack