The art of living with broken things
summary
Claim: end-to-end tests have a different failure policy than unit tests—failing or flaky E2E tests often provide valuable, actionable signal. Built an observability pipeline: results in Elasticsearch, early ElastAlert diffs proved noisy, so added a scikit-learn TF‑IDF classifier tagging individual failures (retrained daily, ~61s, >90% agreement) and a Kibana “scrub/trouble” view over 619 runs. Trend calculator classifies tests as consistently passing/failing, intermittent, flaky, regression/correction (validated by exhaustively testing 2^10 sequences) and teams reuse steps via behave/Robot Framework to make fixes safer.
AI-written summary · about
chapters (10)
- 0:00Introduction and philosophy
- 4:40The value of test failures
- 9:53Scaling and early attempts
- 16:24Moving beyond bulk tracking
- 19:01Categorizing failures with ML
- 26:10Tracking test trends
- 31:04Visualizing actionable data
- 36:03Refining automation practices
- 40:35Frameworks and reuse
- 51:03Conclusion
related talks
46mThe state of the Art in Tackling Flaky Tests
28mTest flakiness; methods for dealing with flaky tests
40mTest trend analysis: towards robust, reliable and timely tests
28mThe State of Art in Tackling Flaky Tests
40mFinding Flakiness: The Real Reasons Your Tests Aren't Reliable
42mTesting on Thin Ice: Chipping Away at Test Unpredictability
44mObservability meets a Flaky Test
01h 15mTaking Control Of Your Test Environment