The State of Art in Tackling Flaky Tests
summary
Industry data shows one in seven tests exhibit flakiness while only ~1.5% of runs fail frequently, and GitHub finds ~24% of tests fail at least once but <1% fail >100 times. Practical responses cluster into keeping builds green, measuring flakiness, and escalation: common tactics are retries (Gradle/Jenkins plugins, VM reshuffle), quarantine and automated reruns (Dropbox), impact scoring and bisecting (GitHub), Bayesian inference to score flakiness (Facebook), and architectural fixes like testable mobile frameworks.
AI-written summary · about
chapters (17)
- 0:00Start
- 2:56Know your enemy
- 3:35Well known causes
- 4:38Flakiness is unevenly distributed
- 6:37Big picture
- 8:10Retries, retries, retries!
- 8:33Not all retries are created equal
- 9:37Cost of retries
- 10:40Ignore flaky tests
- 11:19Psychological barrier
- 13:06Really naive approach
- 13:40Baseline
- 15:12State of art
- 18:30Naive solution
- 22:14Key question
- 22:49Architectural approach
- 25:06Key capability to buy
related talks
46mThe state of the Art in Tackling Flaky Tests
28mTest flakiness; methods for dealing with flaky tests
47mTesting on Thin Ice: Chipping Away at Test Unpredictability
40mFinding Flakiness: The Real Reasons Your Tests Aren't Reliable
42mFlaky tests are a pain
31mCI and the Great Flakiness Adventure
01h 15mTaking Control Of Your Test Environment
16mA Large-Scale Longitudinal Study of Flaky Tests