Zap the Flakes! Leveraging AI to Combat Flaky Tests with CANNIER
summary
Flaky end-to-end tests are flagged by a model that predicts flakiness from code and runtime metrics, so CI reruns target only suspicious tests instead of everything; a Ginkgo-based prototype extracts static and runtime features, runs changed tests only, and exposes predictions from a deployed model service. The approach uses a random forest on static+runtime features, relies on 88% detection within five reruns from prior work, and is implemented as a Go prototype integrating with Ginkgo while feature extraction and storage are still being finished.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 3:18Flaky tests defined
- 7:21Flaky tests impact
- 13:03Detection limits
- 16:16CANNIER approach
- 21:53CANNIER benefits
- 23:45CANNIER implementation
- 24:48Q&A
related talks
41m0Flake - Reaching reliable non-flaky tests
31mCI and the Great Flakiness Adventure
46mThe state of the Art in Tackling Flaky Tests
38mSquash the Flakes! – How to Minimize the Impact of Flaky Tests
28mThe State of Art in Tackling Flaky Tests
28mTest flakiness; methods for dealing with flaky tests
16mA Large-Scale Longitudinal Study of Flaky Tests
47mTesting on Thin Ice: Chipping Away at Test Unpredictability