Regression Testing AI Agents Without Testing the Wrong Thing

QA engineer regression testing AI agents, comparing three agent tool-call sequences on a monitor and spotting the run with a step missing
In this post
  1. One refund, three paths
  2. Your pass-or-fail rule is the problem
  3. Regression testing AI agents: write the rules that cannot change
  4. Where a second AI model helps
  5. Missing evidence is not a pass
  6. Better evidence still needs a testing rule
  7. Build one useful gate this week
  8. Where Nopmark fits
  9. FAQ

Monday's AI agent passes its test.

Tuesday, it does the same job correctly and fails.

Wednesday, it skips a safety check and passes.

That is the trap in regression testing AI agents. The test can reject a harmless change while missing the defect that matters.

A tester described this problem on r/softwaretesting on 25 August 2026. Their agent could reach the right answer through three different tool paths. Exact checks failed good runs. A second AI model, asked to judge the whole run, let bad runs through.

The easiest way to see the problem is through a refund agent.

One refund, three paths

Picture an AI support agent that handles refunds. A person gives it a request in plain language. The agent then decides which tools to use and when to use them.

It can look up a customer, find an order, check whether a refund is allowed, send the money, and add a note to the account.

On Monday, it does this:

  1. Find the customer.
  2. Find the order.
  3. Check permission.
  4. Send the refund.
  5. Add the account note.

The team records that run. The next test expects the same five calls in the same order.

On Tuesday, the agent finds the order before it finds the customer. It still checks permission before sending the money. The correct customer gets the correct refund. The note is right.

The build turns red because the first two calls changed places. Nothing important broke.

On Wednesday, the agent finds the customer and the order, sends the refund, and adds the note. It never checks permission. The money and note look exactly like Monday's result.

An end-result check turns green. Something important did break.

The test got both runs wrong. It cared about the order of two harmless lookups. It did not care whether anyone approved the refund.

Your pass-or-fail rule is the problem

A test needs a rule that decides whether a result is right. Testers call that rule a test oracle. The name sounds academic. The idea is simple: what evidence will make this test pass or fail?

The Monday recording is one possible rule. It says, "A good run must look exactly like this run." That is too strict. It turns choices into requirements even when nobody cares about them.

Checking only the refund is another possible rule. It says, "A good run ends with the right money in the right place." That is too loose. It cannot see the skipped permission check.

The useful rule sits between those two. Do not make the agent repeat Monday's route. Describe what every safe route must prove.

Regression testing AI agents: write the rules that cannot change

For the refund agent, an accepted run must meet four kinds of rule.

1. Check the final result

The refund went to the right customer. The amount matches the order. The account has one refund, not two. The note names the correct order.

These are normal checks against a database or an API. They do not become less useful because an AI agent did the work.

2. Check the required steps

The agent checked permission for this order and this amount. The answer was yes. The check finished before the refund was sent.

Order matters here because the business rule says it matters. A permission check after the refund is not protection. A check for a different order proves nothing.

This does not mean every tool call needs a fixed position. Finding the order before the customer may be fine. Fix the order only where the rule depends on it.

3. Check the forbidden actions

The agent did not send two refunds. It did not change a live customer record during a test. It did not use a tool that this task forbids.

Start this list with the people who own the risk. Ask them, "What could this agent do that would make a correct-looking result unsafe?"

4. Check how much work it used

Suppose good runs need five to eight tool calls. A run that needs 60 calls has probably become stuck, even if it finally reaches the right answer.

Use several good runs to set a starting limit for calls and time. Leave some room. Then watch the results and adjust the limit.

Together, those four checks allow many harmless routes while rejecting unsafe shortcuts.

Where a second AI model helps

Teams often ask another AI model to review the run. This second model is called a judge.

A judge is useful when meaning matters and a simple check cannot settle it. For example: Does the account note explain why the refund was given? Does it name the order in a way a support worker can understand?

Do not ask the judge whether the whole run "looks good." The refund amount, customer ID, permission result, and call order are facts. Check them with code. A model's opinion is weaker than a direct comparison.

Test the judge too. Give it notes that people have already marked as good or bad. Include notes that leave out the refund reason or name the wrong order. If the judge passes those, the judge is part of the bug.

Missing evidence is not a pass

There may be a run where the final refund is correct, but the log does not show whether permission was checked.

Do not guess. The honest result is "unknown" or "blocked." You need better logging before you can approve that run.

This matters most for safety and money. If a required check is missing from the record, missing evidence is not proof that it happened.

Not every missing field needs to stop the build. A missing timing value may be a logging problem. A missing permission record is different.

Better evidence still needs a testing rule

The same week the Reddit thread appeared, Cypress released Cypress Tap. It lets an AI coding agent work with a live Cypress session from the terminal. The agent can run tests, read the output, and inspect the application under test.

Cypress says this lets the agent verify its work instead of reporting only a pass or fail. That is useful because a green message from the tool that did the work is still only a claim.

Cypress Tap gives the agent better evidence about what happened. It does not decide what should have happened. That rule still has to come from the tester and the people who understand the product risk.

Build one useful gate this week

Do not start by solving every agent task. Pick one important job, such as issuing a refund, booking an appointment, or changing an account.

Run the same request several times. Keep the safe runs even when their tool order changes. Then create or find a bad run that reaches the right final result by skipping a required step.

Write down:

  • What must be true at the end.
  • What must happen before the end.
  • What must never happen.
  • How many calls or how much time is too much.
  • What cannot be decided without a human or model judge.

Turn the first four items into direct checks. Give the judge only the last item.

That is the real change in regression testing AI agents. You are no longer freezing one successful route. You are drawing the fence around every route you are willing to accept.

Where Nopmark fits

This is test design. You decide what evidence proves the system worked and what evidence exposes a defect.

Nopmark's ISTQB Certified Tester AI Testing with Generative AI course goes further into using generative AI for test cases, test data, automation scripts, and test oracles. It also covers hallucinations and reasoning errors in model output. See the course at nopmark.com/pages/istqb-ct-genai.

If you do one thing Monday morning, write the forbidden actions and required steps for one agent task. That list will catch more useful failures than a recording of one perfect run.

FAQ

Should I stop asserting on tool calls entirely? No. Check the calls that prove a requirement. A permission check must cover the same refund, finish before the money is sent, and return yes. Do not lock every harmless lookup into one order.

How do I set the cost limit? Count the calls and time used by several good runs. Set the first limit above that range, then adjust it as you collect more results.

Is a model judge ever enough on its own? Only for a narrow question about meaning, such as whether a note explains the refund clearly. Check IDs, amounts, tool results, and required order with code.

My agent's logs do not show tool calls clearly. Where do I start? Start with the final result, but label the test as incomplete. If a skipped tool call could cause real harm, better logging is part of the test work.

Does this apply to agents that write tests, or agents under test? Both. An agent under test needs limits on its actions. An agent that writes tests needs its work checked independently. In both cases, its own pass message is not enough.

Keep reading