Testing AI-Generated Code: 644 Passed, 221 Missed Review Rules

Illustrative scene of a software tester reviewing passing and failing checks while testing AI-generated code.
In this post
  1. The new feature changed something existing code could use
  2. The researchers checked the fix and the rule separately
  3. Turn the review concern into a test
  4. Check that the test can detect the violation
  5. Use the next review comment to choose a check

What does a passing test tell you about an AI fix? In SWE-Gate, 644 fixes passed functional tests, but 221 failed separate checks for review rules. The agents had received those rules in their instructions.

For a tester, the question is how a fix can meet the tested behavior and still miss a stated requirement. The paper introduces that problem through a real code review for Pydantic, a Python library.

The new feature changed something existing code could use

Pydantic pull request #12657 proposed a way to leave certain extra fields out of exported data when a condition was met. The first design added a new structure to describe those fields. That also changed a structure exposed to other code.

Reviewer David Hewitt flagged the risk: code that relied on the existing structure could break. He suggested adding a new option while preserving the old structure. The contributor then revised the change to remove the breaking class. The review discussion records both the concern and the proposed approach.

The review added a requirement that a test of the new output could miss. The change needed to omit the right fields and keep existing uses working. Those are different things to check, even when they concern the same change.

A test that exercises only the new way of exporting data might pass while an existing caller fails. To test compatibility, you need to exercise the interface in a way that existing code still uses. If you rewrite that test to use the new interface, you may remove the very check you needed.

Expected test outcomes.

These are proposed checks based on the Pydantic review, not actual test results. A test that uses the old interface should catch the break.

This review is the paper's opening example. The benchmark then asks how coding agents handle rules drawn from reviews like it.

The researchers checked the fix and the rule separately

The authors built 303 repair tasks across 75 Python projects, creating new scenarios from real review comments. Each task had automated checks for the reported bug and for a review rule. These were constructed tasks, rather than replays of the original pull requests. SWE-Gate methods

Across 4 models, 423 of the 644 fixes passed both sets of checks. The remaining 221 represent 34.3% of the functional passes. Automated tests detected those failures; the original reviewers did not assess the agents' fixes. These figures describe the benchmark, not your team's expected failure rate.

SWE-Gate results.

All 644 fixes passed the tests for the bug. But 221 failed the checks for the stated rules. Source: SWE-Gate, Table 3.

That changes how I would read a report that says an agent's fix passed. I would want to see which requirements were checked and which results support them. A pass on the test for the reported bug gives evidence about that behavior. It does not settle whether the change meets every other rule in the task.

Turn the review concern into a test

For a change like the Pydantic proposal, I would keep the new behavior and the existing interface visible in the test plan. Start by writing down what each needs to do, then choose a test that could reveal a mistake.

The test for the new feature should check which fields appear in the output. Include a case where the condition calls for a field to be omitted. Also include a case where the field should remain, so removing every field cannot produce a passing result.

The test for existing uses should exercise the old interface without changing the calling code to suit the new design. Check that the call still works and produces the expected result. A check that only confirms a name still exists would miss a change to the data that the caller receives.

These are proposed tests for the requirements raised by the review. They are not a reconstruction of Pydantic's test suite.

So what would make the compatibility test fail? Would it detect the change the reviewer objected to? Work through that question before relying on the test. The expected result should follow from the requirement to preserve existing uses.

An agent can help draft the test, but review its inputs and expected results against that requirement. If it changes both the interface and the test to match, they may agree while existing code remains broken. Keep a record of what the old caller expects so that you can spot that change during review.

Check that the test can detect the violation

Once the check is written, try it against the change that violates the rule in an isolated test branch. Inspect the failure to confirm that it came from the behavior you meant to check. A setup error would not prove that the test caught a compatibility problem.

Then try the version that preserves the required behavior. The check should pass for that version, alongside the tests for the new feature. You have now exercised the test against both outcomes instead of seeing it only pass.

That check still covers a specific use of the interface. Tailor the other cases to the ways your product uses it and the risks of breaking those uses. A single passing example does not cover every caller, input or workflow.

Use the next review comment to choose a check

The Pydantic reviewer identified a rule about preserving an existing interface. On your own project, start with a review comment that names a behavior the change must keep. Confirm that the rule still applies, then look for a test that checks it.

If the check is missing, write it from the requirement and show that it detects a violation. Put the rule in the agent's task as well. Keep both the bug-fix results and the rule-check results available to the reviewer when the patch is ready.

The new feature and the existing caller each need evidence that they work. Before accepting an AI fix, follow a review requirement through to the test that checks it. If you cannot find that test, record how the requirement will be checked before merging the change.

Keep reading