Network Observability

on YouTube ↗41m

" summary

Erlang-based InFlow collector parses sFlow and IPFIX via binary pattern matching, runs gen_udp receivers that hand work to parser workers, and handles template state by broadcasting IPFIX templates over a PG group. Flows route to Island shards using P_hash2 sharding into per-key edge tables, are aggregated per minute with sample aggregator hibernation to save memory, and publish via Hackney to Kafka. System processes about 800k datagrams/sec (~5M flows/sec) and emits ~3M aggregated messages/sec.

AI-written summary · about

" chapters (5)

1. 0:0002:02 - Intro and Agenda
2. 2:0307:15 - Intro to InFlow
3. 7:1611:15 - Applications of Flow Data
4. 11:1622:18 - InFlow Collector Architecture
5. 22:1941:22 - InFlow Collector at Scale

" related talks

• 26mNetwork Observability with IPFIX, Prometheus and Elastic StackRastislav Szabo · KubeCon + CloudNativeCon Europe 2019
• 49mRelational Observability for Cloud-Native Security and Data ScienceFrederico Araujo, Teryl Taylor · Open AI + Data Forum @ Open Source Summit North America 2023
• 16mNetwork Flow Data in the CloudSteve Dodd · SREcon25 Americas
• 35mNetwork Flow Monitoring in Kubernetes with Contiv/VPP CNI and Elastic StackRastislav Szabo · KubeCon + CloudNativeCon + Open Source Summit China 2019
• 39mManaging (..) networking flows using ErlangPeer Stritzinger, Sébastien Merle · Code BEAM V Europe 2021
• 23mNetwork Flow Aggregation: Pay for the Logs You Care About!Anubhab Majumdar, Mereta Degutyte, Vamsi Kalapala · CiliumCon Europe 2026
• 28mEasy logging refinement with FlowGDavid Delassus · Open Source Monitoring Conference 2025
• 35mNetwork Flow Monitoring in Kubernetes with Contiv/VPP CNI and the Elastic StackRastislav Szabo · KubeCon + CloudNativeCon Europe 2019