Old Apps, New Tricks: How AI can write Automated Tests for your Shiny Apps
summary
Automatic generation of regression tests for existing Shiny apps targets Shiny for Python only and produces Playwright-based tests that call the framework’s uses Playwright controllers to interact with inputs. The CLI (shiny add test v1.5 on PyPI) sends app code to an LLM via Chatlas-compatible providers (OpenAI, Anthropic), runs an evaluation suite across 10 benchmark apps requiring >80% pass via InspectAI to reduce hallucinations, and typically generates tests in ~6 seconds at about $0.07; tests cover only Shiny components and need stable IDs.
AI-written summary · about
chapters (14)
- 0:00AI tests for Shiny apps
- 4:39Bugs threaten Shiny stability
- 5:01Shiny testing state
- 7:49Evolution of Shiny add test
- 9:14Integrating AI with Shiny add test
- 11:02Why Shiny add test beats AI
- 13:11Handling AI hallucinations
- 14:59Demo: AI test generation
- 16:23Best practices for AI tests
- 17:28Summary
- 17:52Q&A
- 18:14Manual test writing
- 18:55Future of AI tests for R Shiny
- 19:38Azure and Databricks support
related talks
15mA Robust Framework for Automated Shiny App Testing- 15mA Robust Framework for Automated Shiny App Testing
20mBuilding ML and AI apps with Shiny for Python
39mHow to Use AI with Playwright for UI & API Testing
43mExploratory Testing with AI
35mApp Testing with AI Tools – Make it Easy
18m{shinytest2}: Unit testing for Shiny applications
33mShray Sharma: Leveraging AI for Self-Healing Tests: Current Capabilities and Future Prospects