From Intuition to AI: Automating Investigation of Flaky Tests
summary
Describes how Alan turned flaky-test investigation from an expert-only craft into a few-minute task for any engineer by building reproducibility and tooling that are “AI-ready,” then adding an AI agent as the final assist. Covers a pipeline for 27,000 pytest runs on distributed CI, four concrete engineering steps that encoded expert intuition into reliable processes, how to structure tools for reproducibility and AI integration, and the practical role of AI in automating flaky-test debugging.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 0:42Flaky tests at Alan
- 2:07Reproducible failures
- 3:54Documenting instability causes
- 6:24Debug Flaky Test agent
- 9:50Proactive fixes automation
- 11:35Results and key takeaways
- 13:00Conclusion
related talks
44mHow AI is Exposing the Existential Crisis of Software Testing
40mFrom Bugs to Brilliance: Leveraging AI in Modern Software Testing
37mAgentic AI and the Next Decade of Quality Engineering
30mArchitecting Integrity: Responsible Software Evolution in the AI Era
32mQuality engineering in the AI age: From testing AI to leveraging GenAI
42mIt's Inevitable, Mr. Anderson: Machines, Humans, and the Future of Testing
50mPanel Discussion: What Can Go Wrong With AI in Testing?
41mHow AI is transforming software testing