Scaling Kernel Production Monitoring @ Meta
summary
Meta monitors millions of servers running bleeding‑edge kernels and collects kernel logs via netconsole into regional netcond listeners, captures full dumps with kdump and analyzes them with orchestrated tools (drgn, lldb) to produce large JSON crash records; thousands of crashes appear daily and are grouped by hashed stack‑trace, normalized to rates per 1,000 hosts, and surfaced in a denormalized kernel health table of ~10 metrics (host crashes, process crashes, eBPF failures, connectivity) to decide release actions while struggling with small‑sample noisy signals.
AI-written summary · about
chapters (11)
- 0:00Introduction and motivation
- 3:34Netconsole and log collection
- 6:03Kernel crash handling with Kdump
- 8:29Alarms and kernel log analysis
- 10:55Crash normalization and metrics
- 14:03LogView for triage and investigation
- 15:33Kernel health metrics overview
- 17:21Performance as release criteria
- 18:26Sentry integration and event grouping
- 20:39Kernel message parsing challenges
- 21:42Interleaved message handling
related talks
31mFrom Kernel To Cloud: Building a Complete Open Source Observ...
29mKubernetes-Defined Monitoring [I]
37mPracticing Linux Crash/Panic Issue on Production and Cloud Server: Using K...
30mMix Kubernetes with Linux Kernel Tracing to Boost the Observability of Your Cluster- 30mMix Kubernetes with Linux Kernel Tracing to Boost the Observability of Your Cluster
26mStateless Linux Kernel Crashdumps
44mThe Dark Art of Container Monitoring
47mDid My Deploy Degrade Production?