The state of the Art in Tackling Flaky Tests
summary
Industry measurements report one in seven tests show some flakiness while only ~1.45% of test runs fail, exposing a skew where a few tests fail repeatedly. Teams use short retries, broader reruns on different VMs, and quarantine state workflows; Google/Dropbox correlate memory and timing, Spotify groups co-failing tests, Facebook applies bayesian inference to score flakiness, and GitHub builds a dashboard with impact scores. Engineering fixes include rearchitecting testable apps (Airbnb) and avoiding heavyweight acceptance suites (Jenkins experience).
AI-written summary · about
chapters (10)
- 0:00Flaky tests problem
- 2:06Flaky tests serious issue
- 8:45Flakiness causes
- 13:40Three pillars tackle flakiness
- 16:05Keeping builds green with retries
- 24:12Measuring flakiness beyond tribal knowledge
- 31:34Escalating flakiness to developers
- 38:02Architectural solutions reduce flakiness
- 41:51Future industry-wide flakiness solutions
- 42:55Q&A
related talks
28mThe State of Art in Tackling Flaky Tests
28mTest flakiness; methods for dealing with flaky tests
47mTesting on Thin Ice: Chipping Away at Test Unpredictability
40mFinding Flakiness: The Real Reasons Your Tests Aren't Reliable
41m0Flake - Reaching reliable non-flaky tests
39mTen things you heard about testing that might be wrong
42mTesting on Thin Ice: Chipping Away at Test Unpredictability
15mHow to (not) write a flaky test