Passing tests are not necessarily good tests.

First, the intuition.

A test can be perfectly valid Python or JavaScript and always pass because it never checks anything meaningful. Mutation testing asks a deeper question: would the test catch a deliberately introduced bug?

What the researchers actually did

Req2Test-Bench compares requirement-to-test generation across a large repository-derived dataset. It runs generated tests against implementations and injected faulty variants, including a strict measure relative to developer-authored tests.

What the results suggest

• The highest reported raw passing-test result is 82.8%, but only 33.8% meets the stricter kill-complete adequacy measure.
• Increasing agent expenditure does not reliably produce proportional improvements in behavioral adequacy.
• The benchmark illustrates a large gap between syntactic validity and actual fault-detection power.

How convincing is the evidence?

Executable tests, implementations, and fault mutations give an objective check of test quality beyond an LLM judge or test-count metric.

What remains uncertain?

Mutation coverage is a proxy: seeded faults are not all possible real faults. The repository and requirements selection may favor certain frameworks and testing styles.

Connections worth exploring

• Mutation testing is a practical application of falsification.
• Independent or adversarial tests protect against a model grading its own work.
• Coverage and passing counts should not be confused with requirement adequacy.