Better Integration Tests for Performance Monitoring
summary
Slack builds Slerf to detect performance regressions by instrumenting integration tests and CI rather than relying on flaky timing in QA environments. It starts from a list of top 15 APIs (range: millions to tens of millions calls per day) and evolves to benchmark ~800 benchmarks by diffing DB and memcache query counts between branch and master. Tests use an in-memory MySQL simulator sequel fake and a memcache mock, produce JSON diffs, block merges on regressions, and allow an explicit acknowledge-and-merge override.
AI-written summary · about
chapters (16)
- 0:00Introduction and background
- 1:20Context for performance needs
- 2:48Development challenges
- 5:52Need for performance tools
- 7:39Building the initial tool
- 8:51Identifying key APIs
- 11:53First iteration of Slurve
- 14:32Tool reliability and workflow
- 17:13Results of first version
- 19:47Identifying tool limitations
- 20:54New strategy for Slurve
- 22:16Leveraging test infrastructure
- 24:43Implementing the benchmarks
- 26:54Enhanced reporting and UI
- 28:22New version performance
- 30:08Key takeaways and conclusion
related talks
30mHow to Performance
21mScaling performance at Slack's scale
37mUsing SLOs for Continuous Performance Optimizations of Your K8s Workloads
44mE2E2U: Slack’s journey to developer-driven end-to-end testing
45mHow to Build a Performance Testing Pipeline from Scratch
41mShift Left your Performance Testing
53mEarly Performance Testing: Finding News You Can Use
39mCreating & Scaling Meaningful Performance Tests in CI