How to Review AI-Generated Tests Before You Trust the Green

QA engineer scrutinizing rows of green passing checkmarks on a monitor while reviewing AI-generated tests
In this post
  1. The green checkmark is not the result you care about
  2. A checklist to review AI-generated tests
  3. The anti-patterns an AI reaches for
  4. Where this leaves the tester
  5. Where Nopmark fits
  6. FAQ

A team I know once shipped an averages bug that no test caught. The dashboard averaged how long users spent on an event, and the test data was convenient: every user had the same handful of identical values. So the average always came back equal to the input, and the test passed every single run. The bug was that users with no data got counted as zeros instead of being left out, which quietly dragged every average down. Nobody saw it in testing, verification, or validation. A customer on a large tenant found it in production. That gap, between a test that passes and a test that proves something, is the whole reason you have to review AI-generated tests with real suspicion now that a machine writes them for you.

The test was green the whole time. It was just green about the wrong thing.

That is the exact failure mode you now have to watch for at scale, because the machine writes the test in seconds and the green checkmark shows up before you have thought about what it proves. If you want to review AI-generated tests well, you have to stop reading them as code that runs and start reading them as claims that something is true. A passing test is a claim. Your job is to interrogate the claim.

The green checkmark is not the result you care about

An AI test generator is very good at producing tests that execute and pass. That is the thing it optimizes for, because a passing suite looks like success. What it does not optimize for is whether the assertion inside the test would fail if the feature broke. Those are different targets, and the gap between them is where bugs live.

The averages story is the clean version of this. The test data was chosen to make the test easy to write and easy to pass. Identical values meant the tester never had to reason about the awkward case, the user with nothing recorded. Test data optimized for the writer's convenience is test data optimized to miss bugs, and an AI will reach for the convenient data faster than any human because it has no discomfort to push against.

So the first move when you review AI-generated tests is to ignore the pass. Assume it passes. Now ask what it would take to make it fail, and whether that maps to a real defect.

A checklist to review AI-generated tests

Four questions. Run them per test, in order. Any one that comes back wrong sends the test back.

  • Does it assert the right thing? Find the assertion. If there isn't one, or it only checks that the code ran without throwing, the test proves the function did not crash and nothing more. A surprising share of generated tests are tautologies: they mock a dependency to return a value, call the function, and assert it got that value back. That tests the mock, not your logic.
  • Does it assert at the right layer? A unit test that reaches through three real services to check a formatted date is not a unit test, and it will go flaky the moment one of those services hiccups. An end-to-end test that stubs the database is not proving the integration you needed proven. The AI does not know which layer your risk actually lives at. You do.
  • Does it use the right data? This is the averages trap. Convenient inputs, all-happy-path values, no empty case, no boundary, no user-with-no-data. If every input in the test is the kind of input that behaves, the test cannot catch the input that does not.
  • Does it fail for the right reason? Break the feature on purpose in your head, or in the code for ten seconds, and see if this test would notice. If you cannot construct a change to the production code that turns this test red, the test is decoration.

None of these takes long once it is a habit. The whole pass is maybe ten minutes on a normal pull request, and it front-loads the thinking that the averages team paid for later in a customer escalation.

The anti-patterns an AI reaches for

A few show up often enough to name, so you can spot them without reading every line.

Over-mocking is the loudest one. When a generated test mocks nearly everything it touches, the surface it actually exercises shrinks to almost nothing, and you end up asserting that your mocks return what you told them to return. Tautological assertions are the cousin of that: expect(result).toBe(mockValue) where mockValue was the thing you injected two lines up.

Then there is testing the framework instead of the app. A generated test that checks a library sorts a list correctly, or that a well-known validation package rejects a bad email, is spending your CI budget verifying code you did not write and that already has its own tests. Delete those. They add runtime and maintenance and catch nothing about your product.

The last one is the assertion that is technically true but strategically empty: checking a status code is 200 without checking the body, checking a list is non-empty without checking what is in it, checking a save happened without checking what got saved. The averages test was this. It confirmed a number came back. It never confirmed the number was right.

Where this leaves the tester

There is a version of the AI-testing story where the tester disappears because the machine writes the tests. That is not the version playing out. What is actually happening on teams right now is that the bottleneck has moved from writing tests to judging them. Anyone can generate a hundred tests before lunch. The scarce skill is the person who can read those hundred tests and tell you which twelve are load-bearing and which eighty-eight are noise that will rot into flakiness and false confidence.

That reading skill is not new. It is the same critical instinct good testers always had, the willingness to interrogate a claim instead of accepting it, the habit of asking where the requirements author or the developer or now the model made an assumption that does not hold. The tool changed. The discipline did not. If anything the discipline matters more, because the volume of plausible-looking, green, worthless tests just went up by an order of magnitude, and something has to stand between that volume and your definition of done.

Where Nopmark fits

Reading AI output critically is a testing skill, not a prompting trick, and it rests on the fundamentals: knowing your test design techniques, your coverage goals, and what a good assertion actually proves. Nopmark's ISTQB Certified Tester AI Testing (CT-GenAI) course is built for exactly this shift, how to use generative AI in your testing and how to test the AI features your team is shipping, taught by accredited instructors with hands-on labs. It is the difference between generating tests and being the person who can tell the good ones from the green ones.

FAQ

Are AI-generated tests reliable? They are as reliable as the assertions inside them, which is often less reliable than they look. The test executing and passing tells you almost nothing about whether it would catch a regression. Review each one against whether it asserts the right thing, at the right layer, with the right data, for the right reason.

How long should reviewing generated tests take? On a normal pull request, ten minutes or so once the four-question habit is set. That is far cheaper than the production escalation a worthless-but-green test invites.

What is the most common problem with AI-generated tests? Over-mocking and tautological assertions, where the test ends up checking that a mock returns the value it was told to return. Close behind is convenient test data with no empty, boundary, or error cases.

Should I delete AI tests that test the framework, not my code? Yes. A test that checks a third-party library does its own job adds CI time and maintenance while catching nothing about your product. Keep the tests that exercise your logic.

Does AI make manual testers unnecessary? No. It moves the work from writing tests to reviewing them, which raises the value of a tester who can read a test critically and judge what it actually proves.

Ready to build the judgment that separates a load-bearing test from a green one? Nopmark's CT-GenAI course covers how to work with generative AI in testing, taught by accredited instructors with hands-on labs.

Keep reading