Explore LLMs and Multimodal Approaches in Test Automation
summary
Claim: LLMs plus multimodal models can augment test automation by analyzing screenshots and generating runnable tests. The demo runs Llama/Lava via AMA locally, sends screenshots to a multimodal LLM, returns a WCAG 2.2-style accessibility audit, then auto-generates screenshot → accessibility audit → Java Selenium test code. It works after prompt tuning and human-in-the-loop checks, warns about hallucinations, and outlines a scalable pipeline with LangChain, RAG, vector DB for contextual retrieval and enterprise-safe customized models.
AI-written summary · about
chapters (11)
- 0:00AI in test automation
- 2:36Traditional vs AI testing
- 5:45LLMs and multimodal AI
- 9:18How LLMs learn and limits
- 12:10Running LLMs locally with Ollama
- 14:17Multimodal LLM for image analysis
- 18:22LLMs in test automation pipelines
- 25:36Retrieval augmented generation for testing
- 27:22Scaling AI testing with LangChain
- 30:34AI testing ethics
- 32:20Q&A
related talks
49mImproving Accessibility through Leveraging Large Language Models (LLMs)
15mLLMs and Test Automation: A New Era of Self Healing
47mValidating LLMs – A True Story
41mHow continuous testing keeps your LLM on track
19mTesting LLM-Powered Applications
38mMaking LLMs reliable - A practical framework
50mHow to Test LLM Agents
43mFrom Entertaining to Useful: LLMs for Test Tooling