AI coding agent rules: which ones become automatic checks

QA lead pointing at an AI coding agent''s change under review: two items marked as automatic checks and one marked for a person to judge
In this post
  1. What the summit report says about long rule files
  2. Which AI coding agent rules can a script decide?
  3. Two rules become checks
  4. What "fixed" has to mean
  5. The person who can stop the merge
  6. How the change ended
  7. Questions about AI coding agent rules
  8. Sources

Take a QA lead whose team uses an AI coding agent to write and fix tests for an invoicing service. A mistake gets through, so someone adds a line to the agent's rule file, which is the set of written instructions the agent reads before it works. Another mistake gets through, and another line goes in. The file keeps growing, the same mistakes keep coming back, and the agent keeps reporting that the bug is fixed. The rules that a script can decide should run as automatic checks, and "fixed" should mean a test that fails on the old code and passes on the new.

What the summit report says about long rule files

TestDevLab, a testing services company, published "Who's Checking? 5 Lessons from the AI Tech Summit 2026" on October 20, 2026. A TestDevLab QA engineer who attended retells talks from several speakers and draws a lesson from each. Every story below is a speaker's account as the article retells it, and Nopmark did not measure any of it.

In the second lesson, the article describes a talk by James Pelton of Zencoder about adding rules to an agent. The article says "The more rules he added, the worse it got". His answer, in the article's words, was that "For each important rule, he wrote a small script that runs automatically before the agent acts". In the article's example, a script scans an email draft for em dashes and blocks the email if it finds one. The article gives only a few example rules, such as "Ask before deleting anything", and none of them is about tests, so the checks in this post are ours.

In the first lesson, the article describes a talk by Alistair Greenwood of OmniforceAI about an agent that kept saying a bug was fixed while it was not. The article sums up the point in one line: "An agent’s confidence is not evidence, but a reproducible test is".

Which AI coding agent rules can a script decide?

Back to the invoicing team, whose rule file holds three rules that matter here. An assertion is the check a test makes on a result, such as "the invoice total is 120.00". A skipped test is one marked so that the test runner does not run it. An expected value is the result the test treats as correct.

Sort each rule with one question: can a script decide this from the change alone? The change is the set of edits the agent proposes.

  • Every test must assert on the result: yes. A script can read each new test and fail the change when a test has no assertion.
  • Do not skip a failing test: yes. A script can compare the list of tests before and after, and fail the change when a test is newly marked skipped.
  • Never change an expected value just to make a test pass: only in part. A script can see that an expected value changed and flag the change for a person. It cannot say whether the new value is right, because that depends on the requirement, and only a person who knows the requirement can decide.

That last point is my view as an instructor, not a measured result. AI tools tend not to know your business rules, such as a finance rule or a clinical workflow, because they were never trained on your product. A rule that needs that knowledge stays with a person.

Three agent rules sorted by one question. Every test must assert on the result: yes, an automatic check. Do not skip a failing test: yes, an automatic check. Never change an expected value just to make a test pass: only in part, so a script flags it and a person decides.

Two rules become checks

Now the agent opens a change with one test that has no assertion and one newly skipped test, and it reports the bug fixed.

With the rule file alone, nothing stops this change, because the agent read the rules and broke them anyway. With two checks that run before any change can be accepted, both would fail, and the message would name the rule each one enforces. The team no longer depends on the agent remembering the rule.

A check is code, so test it too. Give each one a change it should fail, such as a test with no assertion, and confirm that it does fail. A check that has never failed may not be checking anything.

What "fixed" has to mean

When the agent says the bug is fixed, it is claiming that behavior changed, and a claim needs evidence. Say the reported bug is that a discounted invoice shows the full total. The evidence is a test written for that bug, run twice.

On the previous version of the code, the test must fail, because the bug is there. On the agent's new version, the same test must pass. If it passes on both, the test never caught the bug, so a pass on the new version proves nothing. If it fails on both, the bug is still there.

One test for the reported bug, run on the old and new code. Fails, then passes: evidence the bug is fixed. Passes on both: the test never caught the bug. Fails on both: the bug is still there.

The article's lesson fits here: the agent's report is only a statement, while a test that fails and then passes is something the team can rerun and see for itself.

The person who can stop the merge

A merge adds a change to the team's main code. The third rule, about expected values, needs someone who knows the requirement. Say the agent changed an expected invoice total from 120.00 to 118.00 in one test. The check flags it. A reviewer reads the requirement and decides whether 118.00 is correct or whether the agent bent the test to match faulty code.

That only works if the reviewer has real power. The article's third lesson draws on a talk by Darren Goonawardana of Collectiv. Its summary says "A reviewer who can approve but can’t stop anything is not a safeguard". Check it in practice. Open a harmless change, ask the reviewer to block it, and see whether the merge is actually stopped.

A team that uses AI agents in its testing and wants help setting up these controls can talk to Nopmark about team training. The team, its rules and the change in this post are constructed for the example, so no real agent or test run produced them.

How the change ended

The agent's change never reaches the main code, because two checks fail it before anyone reads it. The claim that the bug is fixed has no failing-then-passing test behind it, so the change waits. When the reviewer then sees the changed expected value, that person can say no, and the merge stays stopped. The rule file can stay short, because the checks enforce the rules a script can decide.

The agent's change meets three gates before the main code. Automatic checks fail the test with no assertion and the skipped test. The fixed claim waits for a test that fails, then passes. A reviewer who can block sees the expected total changed from 120.00 to 118.00.

Questions about AI coding agent rules

What is an AI coding agent? It is an AI tool that takes a task, writes or changes code and tests, and reports back. In this post it writes and fixes tests for an invoicing service.

Which rules should become automatic checks? Rules that a script can decide from the change alone, such as a test with no assertion or a newly skipped test. Rules that need knowledge of the requirement stay with a person, and a script can flag the change for that person.

Is the agent's report that a bug is fixed enough? No. The article says an agent's confidence is not evidence. Ask for a test of the bug that fails on the previous version and passes on the new one.

Why does the reviewer need power to block a change? An approval that cannot be refused is only a signature. The reviewer should be able to stop the merge, and the team should confirm that on a harmless change.

Sources

  • TestDevLab, "Who's Checking? 5 Lessons from the AI Tech Summit 2026", by Anita Miovska, published October 20, 2026. Source for the talks retold in lessons 1 to 3 and the three quoted lines. TestDevLab is a testing services company, and the report is vendor editorial. Speakers' employers and titles are as the article states them and were not verified.

Keep reading