Flakiness in your tests isn't down to test framework
summary
Claim: test flakiness comes from asynchronous systems and test design, not the framework. Examples: frameworks from Selenium, Puppeteer, Cypress, Playwright all promised no wait libraries then added them; navigating a page can trigger countless async network and rendering steps; a single Firefox commit once triggered ~2,500 compute hours and repositories had >100,000 test files. Concrete fixes: instrument pages (for example an invisible pixel), pick a stable DOM point, collect flaky-rate metrics, and accept the halting problem limits when deciding which tests to keep.
AI-written summary · about
chapters (11)
- 0:00Introduction and background
- 2:13Defining test flakiness
- 3:32Challenges of scale
- 5:52Cognitive biases in testing
- 8:23Software complexity
- 10:57Asynchronicity in testing
- 12:55The problem of waiting
- 16:21Framework limitations
- 19:04The halting problem
- 23:50Overton window in testing
- 26:45Q&A and practical advice
related talks
15mHow to (not) write a flaky test
32mHow to write a flaky test
40mFinding Flakiness: The Real Reasons Your Tests Aren't Reliable
46mFilip Hric: How to Write a Flaky Test Description: How (not) to Write a Flaky Test
50mFlaky tests
15mHow to Fix Flakiness in Tests | Using Time Machines to Debug #FlakyTest
31mYour Tests Aren’t Flaky, You Are!
46mThe state of the Art in Tackling Flaky Tests