Risk-based testing: check your scores against escaped defects
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this

On 5 April 2018, the UK bank TSB received a letter from SABIS. SABIS was the IT company building its new banking platform. The letter said SABIS was confident the platform was ready for migration. It gave early results from the non-functional tests, which check things like speed and load. It also noted that some tests were still to be completed. A proposal from a software testing vendor may be written in the same way.
Those tests finished on 17 April. The migration ran over the weekend of 20 to 22 April. Almost at once, online, phone and mobile banking began to fail, and all 550 branches were affected. In December 2022, the FCA and the PRA fined TSB a combined £48.65 million.
The PRA's final notice (20 December 2022) looked closely at that letter. It said the letter and the related supplier confirmations were, "to some extent, forward looking statements of good intention or expectations, rather than statements of fact". SABIS was not an outside testing firm. It belonged to the same banking group, and its contract covered the design, build and testing of the platform. But the buyer's problem is the same. A proposal tells you what a vendor will do, and you need to know what they have done.
A good proposal sets out the scope, the price, the team size and the schedule. It also shows whether the vendor understood your request at all. Use it to build a short list of two or three vendors.
But almost every sentence in a proposal is about the future. "We will assign experienced testers" and "we will report daily" are promises. They are the same kind of statement the PRA found in the TSB letter. The notice also found that TSB did not formally and comprehensively assess, when entering the arrangement, whether SABIS had "the ability, capacity, resources" to deliver the platform on time. A proposal cannot answer that question for you. Evidence can, and one useful source of evidence is a small piece of paid work on your own system.
A pilot is a small, time-boxed piece of work done before the main contract starts. Pick one feature your team knows well, ideally one with a history of defects. Give the vendor the same access, test environments and documents they would get on the full contract. Then watch three things.
Pay attention to the questions the vendor's testers ask. Good testers interrogate the requirements and the code, and their questions get sharper as the pilot goes on. Weak testers wait to be told what to test.
Ask each vendor for documents from past projects, with client names removed. You can check the sample reports against the Foundation Level syllabus, which lists what they typically contain.
The contract is where each promise in the proposal becomes something you can check.
A vendor may refuse a paid pilot on your system and offer only a demo on its own sample app. That vendor is asking you to trust the proposal. A vendor that reports only a pass rate gives you no way to judge what passed or why. Also watch for readiness statements in the future tense, such as "we expect to complete" or "we are confident". Ask what has been done, who did it, and where the evidence is kept. A team that changes between the pilot and the start of the contract means you are paying for people you have not met.
Your own team still has to read the vendor's reports and judge them, and Nopmark can talk with you about training your team to do that.
The vendor you choose will produce many of the results your release decision depends on. If you choose on promises, you will be judging those results without knowing how they were made. When you have seen the team work on your system and the contract lists what they deliver, you can check each result yourself.
Take a QA lead whose team uses risk-based testing to plan each sprint. Every story gets a risk score, the team tests the high scorers first, and the low scorers…
You have read this