Counting What You Care About in Your Security Data Pipeline
summary
AxoSyslog is a high-performance fork of syslog-ng that replaces CSV blobs with native Prometheus metrics and adds programmable, per-log metric emission via update_metric(). The pipeline language (fterx/filterx) lets users build label-rich counters from parsed fields and enrichments, turning a churned fallback topic of tens of thousands of logs into immediate per-agency, per-host dashboards and alerts and eliminating daily manual inspection. The approach is implemented and running in production.
AI-written summary · about
chapters (10)
- 0:00Security data metrics
- 2:50What is syslog
- 5:18Syslog metrics before Prometheus
- 8:37Transition to Prometheus metrics
- 9:20Syslog metrics limitations
- 11:06Log-driven metrics solution
- 12:57Example metrics
- 15:29Visualizing with Sankey diagrams
- 18:40City IT case study
- 22:21Key takeaways
related talks
30mBuilding a Large-Scale, Streaming-Based Logging and Monitoring Solution
33mSeeing the Forest for the Trees: An Operator’s Guide to Observability of CF with...
31mMy Logs Cost More Than My Services
40mObservability that scales: Analyze data in stream and query remotely
22mFluent Bit - Swiss Army Tool of Observability Data Ingestion
49mRelational Observability for Cloud-Native Security and Data Science
27mObservability with Prometheus and beyond
36mObservability with Prometheus and Beyond: Leveraging Cloud-native Technologies Outside of Cloud