Vision: Bias in Systematic Grammar-Based Test Suite Construction Algorithms
summary
Shows that fixed choices in grammar-based test-suite construction create measurable equivalent choice bias and embedding bias that skew generated suites toward certain grammar elements (e.g., sleep vs return). Measures bias by shuffling rule order and switching embedding strategies (shortest vs shallowest yield), then compares line coverage on real systems including SQLite. Finds a worse algorithm can hit 63.5% max coverage while a better one drops to 62.9% (48 vs 429 tests), so algorithmic tie-breakers materially affect coverage.
AI-written summary · about
chapters (6)
related talks
15mGrammar-Based Testing for Little Languages: An Experience Report with Student Compilers
22mThe Good, the Bad and the Ugly solutions in test design
45mUsing AI, statistics, and set theory to know what to test
33mPath to a more equitable testing process: Addressing unconscious bias and ethics
44mHow to Stop Hating your Test Suite
28mLanguage Support for Property-Based Testing
49mHow to Stop Hating Your Test Suite
19mCovering All the Bases: Type-Based Verification of Test Input Generators