A Case Study in Chaos Testing: Uncovering Kernel Scaling Issues
summary
Chaos testing replicates a sporadic production scaling failure and pins it to /proc read paths and thread traversal in a 3.10 kernel; experiments use internal orchestrator Chaos Commander with executor General Disarray, a background idle-thread generator and /proc kernel profiling, and repeat runs reproduce the failure. Expected scale near 4 million PIDs instead breaks around ~300,000 threads, so mitigations include PID namespaces (containers), RCU fanout tuning, nproc-aware load balancing, smaller thread pools and per-task/host monitoring.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 0:16Agenda
- 1:54New clusters
- 2:06Scaling issues
- 3:05Missing observations
- 3:29Metrics
- 4:04Resource Usage
- 6:09Chaos Commander
- 7:02Running the experiment
- 8:11The results
- 9:04Summary
- 9:51What is proc
- 10:33proc PID reader
- 11:22Profiling
- 13:04More Kernel Stuff
- 14:50Paulie McKenney quote
- 15:53Reading the data
- 17:06Summary of results
- 17:51Mitigation efforts
- 19:21Other solutions
- 21:35Conclusion
- 23:09Thanks
related talks
28mExtreme OS Kernel Testing
35mOpening the Box: Diagnosing Operating-System Task-Scheduler...
19mDo nothing fast: How to scale idle cpus?
49mScaling compiled applications
49mScaling compiled applications
52mLinux Kernel Debugging: Going Beyond Printk Messages
27mDebugging Linux Issues with eBPF
16mXkernel: Principled Performance Tunability of Operating System Kernels