Finding Flakiness: The Real Reasons Your Tests Aren't Reliable
summary
Claims that Selenium and UI tests can be made reliable and that flaky tests usually trace to environment or engineering problems rather than magic; demonstrates categories with concrete examples: 10,000 tests 100% green as a feasibility claim, a built test runner platform with logging and monitoring that scales, and failures from a load balancer killing 30s requests and a test harness port pool causing collisions. Each category links to remedies like waits vs event waits, version matching, isolated test data, and joining test logs to system logs.
AI-written summary · about
chapters (20)
- 0:00Test flakiness challenges
- 1:07Why flaky tests matter
- 2:55Myths about flaky tests
- 5:25System not ready for testing
- 6:50Poor UI selectors
- 7:32Temporal defects
- 8:34Flaky system under test
- 9:36Flaky test harness
- 10:40Network issues
- 13:34Test dependencies risks
- 16:47Poor test data
- 18:13Human interference
- 21:04Version mismatches
- 24:17Bad waits/timeouts
- 27:27Load on test runners
- 29:55Misunderstanding and mishandling exceptions
- 33:09Tests too late
- 34:53Solutions for flaky tests
- 36:40Productionizing tests
- 38:25Joining test logs
related talks
31mYour Tests Aren’t Flaky, You Are!
01h 09mFlaky Tests: The Method
48mFlaky Tests 2.0
42mFlaky tests are a pain
47mHow to Un-Flake Flaky Tests- a New Hire's Toolkit
29mCommon Mistakes that make test cases flaky and how to avoid them
46mThe state of the Art in Tackling Flaky Tests
41m0Flake - Reaching reliable non-flaky tests