Lean QA: Automation, Monitoring, Experiments
summary
Describes a QA approach for a ~20-team company with ~150M users that deploys to production every 8 minutes and runs about 200 A/B tests daily (~1,500/month). Covers tooling: an internal Petri framework and a Genetic selector for feature toggles and rollouts, plus automated testing from unit to visual screenshot tests, synthetic monitoring and BI event checks. Explains experiment practices (cohort targeting, gradual rollout and rollback) and gives an example where a UI placement A/B test reduced conversions until the purchase link was moved.
AI-written summary · about
chapters (28)
- 0:00Welcome
- 1:13About Wix
- 2:59Testing scale at Wix
- 4:02Production testing techniques
- 5:50A/B test example
- 6:58What is A/B testing
- 7:41A/B test for retirees
- 9:06Monitoring experiments
- 10:35A/B testing challenges
- 11:40Defense principle in testing
- 13:04Audience targeting in experiments
- 14:08Feature flags for testing
- 17:21Experiment automation
- 18:27Experiment lifecycle
- 20:14Automation at all levels
- 21:42Event analytics
- 23:27A/B test report example
- 24:53Lean testing principles
- 26:44People as key factor
- 27:50Developers in testing
- 28:56Rollback types
- 29:43Process optimization
- 31:06Process flexibility
- 31:49Team tools
- 32:58Support and quality
- 33:39What is Lean QA
- 36:10Quality processes
- 36:50Q&A
related talks
46mQA to QE: Scaling Quality with Ownership, Code, and Culture
36mWhy a Growing Company Needs a Growing QA Team
31mScaling Quality in a Decision Intelligence Platform: The Agentic QA Playbook
39mHow To Elevate QA In Your Organization
21mEstablishing Quality Assurance in Startup environment
53mWhat Happens When Everyone is QA? And I Mean Everyone!
56mNo QA resources? You can still deliver a quality website
48mWeb — The Magic of QA Processes in (Super) High-Load Systems