Squash the Flakes! – How to Minimize the Impact of Flaky Tests
summary
Project runs ~1.5k E2E tests that spawn a full cluster using about 29GB and 14 nodes per run. Team auto-runs changed tests five repeated presubmit runs in random order to catch order-dependent flakes, tags failing cases with a quarantine label so they skip presubmit but still run in periodics, and uses a heatmap plus thresholds (5% over 14 days or 25% over 2 days) to decide action. Result: retests down to <1 on average and much lower CI load.
AI-written summary · about
chapters (9)
- 0:00Introduction to flaky tests
- 2:56What is a flaky test
- 8:16Impact of flaky tests
- 13:42Quarantining flaky tests
- 17:37Tools for flake management
- 24:03Preventing flakes in PRs
- 28:23Key sources of flakiness
- 30:37Stabilizing CI environment
- 33:07Future plans and Q&A
related talks
31mTests Are What You Want, What You Really Really Want
28mTest flakiness; methods for dealing with flaky tests
26mKeeping Your Tests Lean
46mThe state of the Art in Tackling Flaky Tests
41m0Flake - Reaching reliable non-flaky tests
28mThe State of Art in Tackling Flaky Tests
34mZap the Flakes! Leveraging AI to Combat Flaky Tests with CANNIER
47mTesting on Thin Ice: Chipping Away at Test Unpredictability