Risk-based testing: check your scores against escaped defects
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this

A test that went red overnight passes this morning, and nothing in the build changed. How do you tell your team whether that run found a defect? Non-deterministic testing starts by treating one run as a single sample of the system's behaviour.
Testing has always leaned on one property of deterministic software. The same input against the same build returns the same output, every time. That property is what let a single test run stand in for the whole system.
You ran the test once, and it passed or it failed. That result was the system's behaviour rather than a sample of it, and you could act on it.
Now a growing share of what we test does not work that way. A model can answer the same question twice and word it two ways. A retrieval step can rank two documents in a new order. An agent can take four steps on Monday and six on Tuesday to reach the same place. In each case the system is doing what it was built to do.
The one-run reflex assumes otherwise, so you assert exact equality, you get one result, and you treat that result as the answer.
Jason Arbon put a simple procedure on this in TestGuild episode 600, recorded on 11 August 2026. Run the suite five times against the same build. The technique needs no new tooling, no vendor and no model.
Five runs give you a distribution instead of a data point. Five results against an unchanged build show how much the system moves on its own. Anything inside that movement is not something you can write an exact assertion against. Anything outside it is worth looking at.
There is nothing special about five, beyond running cheaply on a laptop and being enough to separate "this failed once" from "this fails". Use seven if the suite is cheap. Use three if it is expensive and you accept a blunter read.
There are 3 patterns you will see, and they need different responses.
Those last two are the hard ones to separate. A useful question to interrogate the mixed case: does the output differ, or does only the timing differ? Timing that varies while the output stays the same points at your test framework. Output that differs on the same input points at the system.
Teams get stuck when they find real variance and try to remove it. They set a temperature to zero, pin a seed, freeze a model version, and get their old equality assertions working again.
That may be the right call for a narrow unit test. As a strategy for the feature, it is wasteful. You have built a test setup that behaves nothing like production, and you have switched off the very behaviour the feature needs checked.
The other option is to move from a point to a range. Instead of asking whether the output equals this exact value, you ask whether it stays inside the bounds you set. You apply that across a set of inputs you chose on purpose.
A good input set includes the ordinary cases, the rare ones, and the ones your product manager would be unhappy to see mishandled. Equivalence partitioning and boundary value analysis do not stop working because the system got less certain.
What changes is the oracle, because you are no longer comparing to a stored answer. You are checking a property instead. The answer cites a real document, the answer stays under 200 words, the answer never suggests a dose, the classification falls in the right partition. Properties keep holding when the wording moves, which is what a stored answer cannot do.
A tester wrote on r/softwaretesting this month about their team's model output checks. They had hit the ceiling of what a spreadsheet could carry. The process that carried them through seed stage had become the thing slowing them down. Every model change meant a human rereading a few hundred outputs by hand.
That path fails in a predictable way, and the arithmetic is what breaks it. Reading 200 outputs takes a person most of a day. Two model changes a week means two days of reading. Nobody sustains that, so the sample quietly shrinks, and nobody records that it did.
Manual review of model output repeats the manual regression problem. It seems logical and it seems easy, and it means you never find the time to build the thing that would replace it.
The replacement is ordinary work. Take the outputs you have already judged by hand, and turn them into a fixed test set with the expected results written down. Write property checks for the ones a machine can check. Keep human review for the cases that need judgment, and let the machine do the rest. Then run it five times.
Arbon also makes a claim about cost: testing effort, he argues, scales quadratically with code volume. Ten times the code demands roughly a hundred times the testing.
Take that seriously and five runs sounds expensive, because it is. Five runs of every test, on every commit, will make your pipeline unusable by Thursday.
Tailor it to the risk, because five runs belong on the suites that cover non-deterministic behaviour and on the ones that gate a release. Add any test that has already reported a result you could not reproduce. Everything else keeps a single run.
Most of what makes non-deterministic testing tractable is ordinary test design applied to an unusual oracle. You choose a representative input set, partition it, find the boundaries, and decide what "correct" means before you run anything. Foundation Level covers all of it, and none of it expires when the system stops being predictable. Nopmark's ISTQB CTFL course teaches those techniques as engineering discipline you apply on the job. For teams working directly on model-backed features, the ISTQB CT-GenAI course covers the AI-specific ground on top of it.
Five runs against the same build is a good default, and it comes from Jason Arbon on TestGuild episode 600. Five is small enough to run cheaply and large enough to separate a one-off from a pattern. Use more runs when the suite is fast and the decision matters.
It can be either, and the tell is usually in what varies. If the output is the same and only the timing moves, suspect your test framework. If the output itself differs on the same input, the system is varying and your assertion is probably too tight.
You can, and for a narrow unit test that may be reasonable. As a whole strategy it removes the behaviour you most need to check. Your test setup stops looking like production, and the risk you were trying to cover goes untested.
Assert properties and bounds. The answer cites a source that exists. The classification falls in the expected partition. The response stays inside a length or latency limit, and the model never produces a banned category of output. Properties hold up under variance.
No. Most software is still deterministic and your existing suite is still the right tool for it. The five-run rule applies to the parts of the system that can legitimately answer differently twice.
Nopmark's ISTQB Certified Tester Foundation Level course teaches equivalence partitioning, boundary value analysis, and risk-based test planning, with accredited instructors and hands-on practice. Those are the techniques that carry over to systems that answer differently every time.
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this