A Partial Taxonomy of Test Unreliability
summary
A partial taxonomy catalogs concrete causes of unreliable tests with Node core examples: race conditions (deadlocks, too many processes, firewall throttling), range/port overflows (start near ~65k), and bad timing choices like magic number 100 loops or 1000 ms timeouts. Empirical detail: 33 CI platforms and ~1% flakiness makes failures frequent; running tests repeatedly in CI (parallel and sequential jobs) reliably reproduces 1-in-10k failures. Practical fixes: use events/callbacks, remove magic numbers, and add rerun jobs to confirm a real fix.
AI-written summary · about
chapters (5)
- 0:00Start
- 0:48PALACE FAMILY STEAK HOUSE
- 3:23Taxonomy!
- 17:46Conclusions
- 18:06Bonus: Ask Me About Windows!
related talks
40mFinding Flakiness: The Real Reasons Your Tests Aren't Reliable
46mThe state of the Art in Tackling Flaky Tests
42mTesting on Thin Ice: Chipping Away at Test Unpredictability
24mHow to solve intermittent issues, aka Random Failures
15mHow to (not) write a flaky test
29mCommon Mistakes that make test cases flaky and how to avoid them
28mThe State of Art in Tackling Flaky Tests
41mDeterministic Solutions to Intermittent Failures