Backwards Scoring: Ranking Test Suites by Which Real Incidents They Would Have Caught
summary
A method replays each logged incident against the system state just before its first symptom so every test gets a verdict by failure history: Record, Revert, Replay, Tally. Applied to ~4,200 automated tests over 18 months it yields 80% proven, 8% duplicate, 12% unproven. Unproven tests are flagged for owner review not auto deletion; preventive tests and unlogged incidents are known blind spots because they never produce catch history.
AI-written summary · about
chapters (21)
- 0:00Everyone Tracks Code Coverage; Nobody Tracks What a Test Has Ever Caught
- 1:06Speaker Intro - Partha Sarathi Samal, QE Manager at Paramount
- 1:36The Three Verdicts: Proven, Duplicate, Unproven
- 3:10The Case Study: 4,200 Tests Replayed Against 18 Months of Incidents
- 3:41The Numbers: 80% Proven, 8% Duplicate, 12% Unproven
- 4:11Why Coverage Isn't Enough: Two Tests, Same Coverage, Different Value
- 5:15Defining the Method and the Verdicts Properly
- 6:51Cross-Replay Is What Makes It an Evaluation, Not a Retrospective
- 7:21The Four Steps: Record, Revert, Replay, Tally
- 8:23What Each Verdict Actually Means
- 9:28Duplicate vs Unproven: They Look Identical on a Dashboard but Aren't
- 10:00The Preventive Blind Spot: A Test That Stops Bugs Never Scores Proven
- 12:01The Data Quality Limit: An Unlogged Incident Can Never Be Caught
- 13:05Read Every Verdict as a Floor on Value, Never a Ceiling
- 13:35The Workflow: Flags Go to Test Owners, Not to a Delete Script
- 15:07Coverage Thresholds vs Backward Scoring, Side by Side
- 16:42Five Takeaways for Your Own Test Suite
- 20:17How It Differs From Mutation Testing, Test Impact Analysis, Chaos Engineering
- 23:19Coverage Gap vs Unproven Test - Not the Same Finding
- 23:49Mapping Incidents to Tests Deterministically and Handling Flaky Tests
- 27:03Q&A: Adapting Unflagged Tests, Human in the Loop, Review Bottlenecks
