Evaluating the Quality of Student-Written Software Tests with Curated Mutation Analysis
summary
A TypeScript tool called the mutation grader runs student tests against a perfect reference and hand‑written mutants (16 for autograder, ~100 for evaluation); curated Mutation Analysis is feasible for fast feedback. Mutation scores correlate strongly with fault finding (r ≈ 0.81 correlation), students graded by mutants ask more specification questions and write more assertions with greater variety, and cohorts show fewer tests but not faster implementation. An unexpected finding is a higher peak fault score in the coverage cohort versus the first mutant cohort.
AI-written summary · about
chapters (12)
- 0:00Introduction
- 3:53Limitations of coverage metrics
- 8:50Traditional mutation analysis challenges
- 10:12Curated mutation analysis
- 12:59Comparing cohorts and research
- 15:25Evaluating grader accuracy
- 16:08Impact on test quality
- 17:55Specification understanding
- 19:00Implementation velocity
- 20:03Student test practices
- 23:12Summary
- 23:55Q&A
related talks
43mMutation Testing, Finally a Good Measure of Test Quality?
07mWho is Testing the Tests?
37mAre your tests good enough?
26mMutation Testing
30mMutation Testing, Finally a Good Measure of Test Quality?
20mUnleashing the Power of Mutation Testing: Supercharging Software Quality and Resilien...
32mCAN WE WRITE PERFECT TESTS? - MAYBE!
50mMutation Testing in JavaScript