Risk-based testing: check your scores against escaped defects
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this

Take a QA lead whose team uses an AI coding agent to write and fix tests for an invoicing service. A mistake gets through, so someone adds a line to the agent's rule file, which is the set of written instructions the agent reads before it works. Another mistake gets through, and another line goes in. The file keeps growing, the same mistakes keep coming back, and the agent keeps reporting that the bug is fixed. The rules that a script can decide should run as automatic checks, and "fixed" should mean a test that fails on the old code and passes on the new.
TestDevLab, a testing services company, published "Who's Checking? 5 Lessons from the AI Tech Summit 2026" on October 20, 2026. A TestDevLab QA engineer who attended retells talks from several speakers and draws a lesson from each. Every story below is a speaker's account as the article retells it, and Nopmark did not measure any of it.
In the second lesson, the article describes a talk by James Pelton of Zencoder about adding rules to an agent. The article says "The more rules he added, the worse it got". His answer, in the article's words, was that "For each important rule, he wrote a small script that runs automatically before the agent acts". In the article's example, a script scans an email draft for em dashes and blocks the email if it finds one. The article gives only a few example rules, such as "Ask before deleting anything", and none of them is about tests, so the checks in this post are ours.
In the first lesson, the article describes a talk by Alistair Greenwood of OmniforceAI about an agent that kept saying a bug was fixed while it was not. The article sums up the point in one line: "An agent’s confidence is not evidence, but a reproducible test is".
Back to the invoicing team, whose rule file holds three rules that matter here. An assertion is the check a test makes on a result, such as "the invoice total is 120.00". A skipped test is one marked so that the test runner does not run it. An expected value is the result the test treats as correct.
Sort each rule with one question: can a script decide this from the change alone? The change is the set of edits the agent proposes.
That last point is my view as an instructor, not a measured result. AI tools tend not to know your business rules, such as a finance rule or a clinical workflow, because they were never trained on your product. A rule that needs that knowledge stays with a person.

Now the agent opens a change with one test that has no assertion and one newly skipped test, and it reports the bug fixed.
With the rule file alone, nothing stops this change, because the agent read the rules and broke them anyway. With two checks that run before any change can be accepted, both would fail, and the message would name the rule each one enforces. The team no longer depends on the agent remembering the rule.
A check is code, so test it too. Give each one a change it should fail, such as a test with no assertion, and confirm that it does fail. A check that has never failed may not be checking anything.
When the agent says the bug is fixed, it is claiming that behavior changed, and a claim needs evidence. Say the reported bug is that a discounted invoice shows the full total. The evidence is a test written for that bug, run twice.
On the previous version of the code, the test must fail, because the bug is there. On the agent's new version, the same test must pass. If it passes on both, the test never caught the bug, so a pass on the new version proves nothing. If it fails on both, the bug is still there.

The article's lesson fits here: the agent's report is only a statement, while a test that fails and then passes is something the team can rerun and see for itself.
A merge adds a change to the team's main code. The third rule, about expected values, needs someone who knows the requirement. Say the agent changed an expected invoice total from 120.00 to 118.00 in one test. The check flags it. A reviewer reads the requirement and decides whether 118.00 is correct or whether the agent bent the test to match faulty code.
That only works if the reviewer has real power. The article's third lesson draws on a talk by Darren Goonawardana of Collectiv. Its summary says "A reviewer who can approve but can’t stop anything is not a safeguard". Check it in practice. Open a harmless change, ask the reviewer to block it, and see whether the merge is actually stopped.
A team that uses AI agents in its testing and wants help setting up these controls can talk to Nopmark about team training. The team, its rules and the change in this post are constructed for the example, so no real agent or test run produced them.
The agent's change never reaches the main code, because two checks fail it before anyone reads it. The claim that the bug is fixed has no failing-then-passing test behind it, so the change waits. When the reviewer then sees the changed expected value, that person can say no, and the merge stays stopped. The rule file can stay short, because the checks enforce the rules a script can decide.

What is an AI coding agent? It is an AI tool that takes a task, writes or changes code and tests, and reports back. In this post it writes and fixes tests for an invoicing service.
Which rules should become automatic checks? Rules that a script can decide from the change alone, such as a test with no assertion or a newly skipped test. Rules that need knowledge of the requirement stay with a person, and a script can flag the change for that person.
Is the agent's report that a bug is fixed enough? No. The article says an agent's confidence is not evidence. Ask for a test of the bug that fails on the previous version and passes on the new one.
Why does the reviewer need power to block a change? An approval that cannot be refused is only a signature. The reviewer should be able to stop the merge, and the team should confirm that on a harmless change.
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this