Risk-based testing: check your scores against escaped defects
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this

Take a tester with 12 years on a payroll and HR product. Her team adds an assistant to the employee portal. It is built on a large language model (LLM), the kind of AI that writes text answers, and it answers questions about the company's leave policy. She is asked to test it before release, but AI testing skills were never part of her job before. In her first session she asks three questions. Can I take vacation days before I have earned them? Can I cash out my unused sick days? Can I carry all my unused vacation into next year?
In this setup, the written policy says no to all three. The assistant says yes to all three, in friendly and confident language. She sees the pattern quickly. Each question asks "Am I allowed to do this?", and each has the correct answer no. That is one group of inputs that the system handled wrongly in the same way, and she has tested groups like that for 12 years.
Equivalence partitioning splits inputs into groups that the system should treat the same way. The ISTQB Foundation Level syllabus v4.0.1 explains the theory in section 4.2.1. If one value from a partition finds a defect, any other value from that partition should find it too. That is why it says "one test for each partition is sufficient" when you design tests.
That rule may have served you well for years on forms, APIs and batch jobs. It holds only when the system really treats each value in a partition the same way. You can still build partitions for an AI chatbot. For this LLM feature, do not assume one representative input and one execution give enough evidence, for two reasons.
First, an LLM may not give the same answer twice. The ISTQB AI Testing syllabus v1.0 says in section 8.4 that a test may have several valid outcomes from the same inputs. It adds that a tester may need to run a test several times to get a statistically valid result. Second, two questions that mean the same thing are not the same input to the model. Take "Can I cash out my sick days?" and "Will I get paid for sick days I never used?" as an example. They sit in one partition for you, but the model may treat them differently.
So what do you keep? You keep the partitions, and you may find you need to think harder about them. For a business bot, you might start by splitting questions by the correct answer. The groups could be allowed, not allowed, depends on details, and outside the bot's scope. Treat these groups as a first guess. Two rules with the same "no" answer may still be handled in different ways, so you may need to split them later. Then you test each partition with several wordings, and you run each wording more than once. More wordings cover the ways people ask. More runs show whether the same question gets the same answer. Your partitions are now groups of meaning, and your test count goes up inside each one.
When each partition needs more runs, you cannot test everything equally. You have to choose where the runs go, and senior testers have made that choice for years.
The Foundation syllabus, section 5.2.1, describes a risk by two factors. Risk likelihood is how probable the failure is. Risk impact is the harm it causes. Together they give the risk level, and higher levels get more testing. You have probably run that conversation in many planning meetings.
For the leave assistant, impact is not the same across partitions. A wrong answer about where to find a form wastes a few minutes. A wrong "yes" about cashing out sick days may lead an employee to plan around money they will not get, and it may lead to a dispute with HR. In this example, the sick-day payout case looks like the top risk, so it gets the most wordings and repeat runs at first. Check that choice against the other partitions. A wrong "no" that denies an allowed benefit can also cause harm, and likelihood still counts.
On a normal system, you may estimate likelihood from code changes and defect history. On an LLM feature you often cannot guess it well before testing. You may need to sample first, then rank. Run a small set of questions from each partition a few times, say five runs as a first sample. Keep the model, prompt and settings the same. Count wrong answers out of total runs, such as 3 of 20. Then put your extra effort where both the impact and the failure rate are high. A small sample with no failures does not prove a high-impact partition is safe, so keep testing it.
When you have years of defect reports behind you, error guessing also transfers well. The syllabus, in section 4.4.1, bases it on the tester's knowledge, including failures seen in similar applications. Those reports teach you where systems tend to break. Rules with exceptions are one place, and recent rule changes are another. Those are the same places an LLM may give a confident wrong answer.
A test oracle is the source you use to decide whether a result is right. On many systems you may not have had to think hard about it. If the requirement said the total is $42.50, the expected result was $42.50.
The AI Testing syllabus calls the difficulty of finding expected results "the test oracle problem," in section 8.7. It notes that an oracle "may not necessarily provide an expected value" to compare against. It may only give a way to decide whether the system is working correctly. It also suggests limits or tolerances, and consulting an expert, while noting that the expert may still be wrong.
For the leave assistant, there is no single expected sentence. But the company's written leave policy is an oracle you can check against. You can list the facts a correct answer must contain and the claims it must never make. Any answer that says unused sick days can be paid out fails, whatever the wording. That kind of check is something many senior testers have written before, just not under this name.
A team without a written oracle may end up saying the assistant is "mostly right," and nobody can measure that. With one, a tester can report that the assistant gave a wrong "yes" in a stated number of runs. She can name the partition and the policy line it broke.
Your experience does not cover everything, and the three skills below may be new to you.
Do employers pay more for AI skills? For some skills, the data suggests they do. PwC studied nearly one billion job ads for its 2025 Global AI Jobs Barometer, released in June 2025. It reported an average wage premium of 56% for advanced AI skills in 2024. PwC's examples are machine learning and prompt engineering, not testing. Treat that number as a market signal, not a promise about QA salaries.
For a structured path, ISTQB released version 2.0 of its AI Testing syllabus on April 21, 2026. ISTQB says it covers model testing, including metamorphic and back-to-back testing, and techniques for generative AI and LLMs.
If you learned these techniques years ago, the free Nopmark CTFL assessment shows which Foundation topics still need work.
In the payroll example, the three wrong answers form a pattern, and the tester found it with tools she already had. A partition table, a risk ranking and the written leave policy did that work. A senior tester who brings those to an AI feature does not start over. The team then gets a defect report it can act on before employees start planning around wrong answers.
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this