Test flakiness; methods for dealing with flaky tests
summary
Shows concrete fixes: use jest-junit XML output, ingest results into a DB, and flag tests that fail inconsistently; run suspicious tests 50–100 times to measure flakiness or use a rolling seven‑day window to detect instability. Turn E2E calls into recorded responses with Poli to make tests deterministic. Build visualizers (test-by-build heatmaps), run premerge checks with flaky BOTS, and auto-create tickets; these reduced measured flakiness by ~2–3% in two months.
AI-written summary · about
chapters (31)
- 0:00Introduction
- 0:17Shoutout
- 0:28Who am I
- 1:15What is test flakiness
- 1:28Follow John on Twitter
- 2:01Formal definition
- 2:22Cost of flakiness
- 3:12Loss of confidence
- 4:07Spotify
- 5:00Build times
- 5:54Improving build times
- 6:24Demo
- 8:03Common causes of flaky tests
- 8:26Why is this test flaky
- 9:50Inconsistent assertion timing
- 10:11Fake app example
- 10:56Async await test
- 11:34Test order
- 12:44Reset State
- 13:32Endtoend tests
- 13:49Endtoend test
- 14:05Test pyramids
- 15:27Justpreset
- 16:55Spotify system
- 17:56OdinI system
- 18:49flakiness visualization
- 19:42flakybots
- 20:45masterguardian
- 22:15XML files
- 23:16flakeiness detection
- 24:44master method
related talks
46mThe state of the Art in Tackling Flaky Tests
28mThe State of Art in Tackling Flaky Tests
15mHow to (not) write a flaky test
47mTesting on Thin Ice: Chipping Away at Test Unpredictability
42mTesting on Thin Ice: Chipping Away at Test Unpredictability
01h 09mFlaky Tests: The Method
48mFlaky Tests 2.0
40mFinding Flakiness: The Real Reasons Your Tests Aren't Reliable