Risk-based testing: check your scores against escaped defects

QA lead and a developer checking risk-based testing scores: story cards sorted into high, medium and low risk tiers, with red dots marking escaped defects in the low tier
In this post
  1. What risk-based testing asks the team to score
  2. What the release showed the lead's team
  3. Who was missing from the scoring meeting
  4. How to compare escapes with the risk tier
  5. What the lead changed for the next sprint
  6. Questions about risk-based testing
  7. Where Nopmark fits
  8. Sources

Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers get a lighter pass. This sprint the team scored twelve stories in a short meeting with the product owner, and no dev attended. Four stories that change the order total each got a low probability of failure, because the screens look simple. After the release, three of the four defects that escaped to production came from those four stories.

When escaped defects cluster in the stories you rated low, release after release, a scoring input was wrong. Adding tests to every story leaves the scores unchanged.

The team and the counts in this post are made up for teaching, and nothing was measured.

What risk-based testing asks the team to score

The lead's team was using the method that TestRail's guide to risk-based testing, published on November 10, describes. Risk-based testing is a way to decide which parts of a product to test first and how much to test them. The guide defines risk as "Risk = Probability x Impact". Probability is how likely a story is to fail. Impact is how much harm a failure would cause users or the business.

The team rates both, combines them, and puts each story into a risk tier. A risk tier is a group, such as high, medium or low, and each tier gets a different amount of testing. The guide adds that the method has a limit built in: "Despite its focus on high-priority components, RBT doesn't completely ignore lower-risk items." Here RBT means risk-based testing.

TestRail sells test management software, and its guide ends with a product pitch. This post uses only its definition, its advice on checking scores, and one sentence about who should help set them.

Each score is a prediction, because rating a story low says the team expects it to fail rarely or to cause little harm. A prediction can be checked after the release, yet many teams may never check it, because the scores sit in a spreadsheet and can look like facts.

Diagram of a risk score in risk-based testing. Probability times impact gives risk. Stories then go into a high, medium or low risk tier. The high tier is tested first. The low tier gets a lighter pass.

What the release showed the lead's team

After the release, the lead read the list of escaped defects. An escaped defect is a defect that reaches production, where real users meet it, instead of being found in testing. Three of the four came from the stories rated low. All three traced to the same pricing module, which the devs have long called fragile.

Made-up example of one sprint with twelve scored stories. Four stories were rated low. Three of the four escaped defects came from those four stories. All three traced to the same pricing module.

The first idea in the room was to add tests to every story in the next sprint. That costs time in every sprint and leaves the scoring unchanged. The team would again rate the pricing stories low and give them a light pass, so the module that failed would still get the least attention.

For high-risk defects in production, the guide says to "recalibrate your scoring process, not introduce more tests." To recalibrate means to adjust the scoring so it matches what actually happened. That sentence is about high-risk defects. Applying it to escapes from the low tier is my reasoning, and the guide does not say it. The logic holds when escapes cluster in the low tier across releases. That pattern means a scoring input was wrong.

Who was missing from the scoring meeting

Back with the lead's team, the question was why the pricing stories had been rated low. The guide gives a hint in one sentence: "QA teams know where defects live, while developers are familiar with each module's fragility."

In the example, no dev attended the meeting. The testers and the product owner saw simple screens, and they scored the stories on that. The devs knew that the code behind those screens was fragile. A scoring session without devs can rate a fragile module as unlikely to fail, because nobody who knows its history is in the room.

Made-up example of a risk scoring meeting. Testers and the product owner were in the room and saw simple screens. The devs were not in the room, and they knew the code was fragile. The stories got a low probability of failure.

That is my reading of the guide's sentence, and your team may have a different missing input. It might be a customer who knows which prices matter, or a support agent who sees which complaints repeat. In each case you look for the person who knows the module and was never asked.

How to compare escapes with the risk tier

To check the scores, the lead uses the comparison that the guide describes. The guide says to compare defect results against each risk tier after running the tests. It also calls escaped defects by risk tier "the most important metric" of the method, using RBT for risk-based testing.

After each release, the comparison takes four steps.

  • List the defects. Write down each defect found in testing and each defect that escaped, with the story it came from.
  • Add the tier. Next to each defect, write the risk tier its story was given before testing began.
  • Group by tier. Count defects found and defects escaped in each tier, so you see where the surprises were.
  • Read the groups. A cluster of escapes in the low tier points to a scoring input that was wrong, and you change that input before the next scoring session.
Four steps to compare escaped defects with the risk tier after each release. List the defects. Add the tier each story was given. Group the defects by tier. Read the groups, and change the scoring input that was wrong.

One sprint gives few defects, so treat a single comparison as a hint. Keep the list across sprints, and the pattern becomes clearer. The guide gives no minimum count, and any number I chose would be arbitrary.

Check the probability guess, the impact guess and the people who made them. If the pricing stories were rated low because nobody knew how fragile the module was, that is the input to change.

What the lead changed for the next sprint

Instead of adding tests to every story, the lead asked a dev to join the next scoring session. The probability score went up for every story that touches the pricing module, and the team tested those stories earlier and harder than before.

After the next release, the lead made the same list again and grouped the defects by tier. If fewer escapes now come from the low tier, the scores are closer to the truth. If the escapes still cluster there, the team has another input to look for, and it still has not spent a sprint adding tests where they were not needed.

If the team never makes this list, the pricing module keeps its low score and the next release ships with the same blind spot. Customers then find the defects before the testers do.

Questions about risk-based testing

What is a risk tier? It is a group of stories with a similar risk score, such as high, medium and low. Each tier gets a different amount of testing. No scale or cut-off is a standard, and the guide itself mixes a 1 to 5 scale with a 1 to 3 scale. Your team should pick its own and write it down.

Does risk-based testing mean skipping low-risk stories? No. The guide says the method "doesn't completely ignore lower-risk items." Low-tier stories get a lighter pass and still get one. The lighter pass is the reason their escapes are worth checking.

Should devs join the scoring meeting? The guide says QA teams, developers and business stakeholders should all take part in risk scoring. Testers know where defects live, and devs know how fragile each module is. A meeting with only one group may miss part of the input.

How do I know if my scores are wrong? Compare escaped defects with the tier of their stories after each release. Escapes that cluster in the low tier suggest the low ratings were wrong. One release is a hint, and several releases make a pattern.

Where Nopmark fits

Nopmark trains testers and QA teams. For a team that wants its risk-based testing to hold up release after release, a training-needs conversation is the place to start. You can begin one through the Nopmark contact page.

Sources

Keep reading