How to choose a software testing vendor: ask for proof, not promises

How to choose a software testing vendor: two managers review pilot results on a wall screen
In this post
  1. What a vendor proposal can and cannot tell you
  2. Pay for a short pilot before the long contract
  3. Evidence to request from a software testing vendor
  4. Contract terms that turn promises into deliverables
  5. Warning signs before you sign
  6. Sources

On 5 April 2018, the UK bank TSB received a letter from SABIS. SABIS was the IT company building its new banking platform. The letter said SABIS was confident the platform was ready for migration. It gave early results from the non-functional tests, which check things like speed and load. It also noted that some tests were still to be completed. A proposal from a software testing vendor may be written in the same way.

Those tests finished on 17 April. The migration ran over the weekend of 20 to 22 April. Almost at once, online, phone and mobile banking began to fail, and all 550 branches were affected. In December 2022, the FCA and the PRA fined TSB a combined £48.65 million.

The PRA's final notice (20 December 2022) looked closely at that letter. It said the letter and the related supplier confirmations were, "to some extent, forward looking statements of good intention or expectations, rather than statements of fact". SABIS was not an outside testing firm. It belonged to the same banking group, and its contract covered the design, build and testing of the platform. But the buyer's problem is the same. A proposal tells you what a vendor will do, and you need to know what they have done.

What a vendor proposal can and cannot tell you

A good proposal sets out the scope, the price, the team size and the schedule. It also shows whether the vendor understood your request at all. Use it to build a short list of two or three vendors.

But almost every sentence in a proposal is about the future. "We will assign experienced testers" and "we will report daily" are promises. They are the same kind of statement the PRA found in the TSB letter. The notice also found that TSB did not formally and comprehensively assess, when entering the arrangement, whether SABIS had "the ability, capacity, resources" to deliver the platform on time. A proposal cannot answer that question for you. Evidence can, and one useful source of evidence is a small piece of paid work on your own system.

Pay for a short pilot before the long contract

A pilot is a small, time-boxed piece of work done before the main contract starts. Pick one feature your team knows well, ideally one with a history of defects. Give the vendor the same access, test environments and documents they would get on the full contract. Then watch three things.

  • Who did the work. An unpaid pilot may be staffed by the vendor's sales engineers, who may not work on your project. Ask for the names of the testers in the pilot. Then make sure the contract names the same people.
  • How they work with your developers. Section 1.5.3 of the ISTQB Foundation Level syllabus (v4.0.1) covers outside testers. It calls testing by people from outside the organization "very high independence". It says independent testers tend to find different kinds of defects. It also warns that they may be isolated from the developers, with communication problems or an adversarial relationship. A pilot shows you which kind of team you are buying.
  • What they found. Did they find the known defects you confirmed are still in the pilot build and in scope? Could your developers reproduce the new ones from the reports alone? If they missed a known defect, find out why before you judge their depth. The build, test data, access or time allowed may be the cause.

Pay attention to the questions the vendor's testers ask. Good testers interrogate the requirements and the code, and their questions get sharper as the pilot goes on. Weak testers wait to be told what to test.

Evidence to request from a software testing vendor

Ask each vendor for documents from past projects, with client names removed. You can check the sample reports against the Foundation Level syllabus, which lists what they typically contain.

  • Sample defect reports. A defect report is the record a tester writes when the software does not behave as expected. Section 5.5 of the syllabus lists the usual contents. These include steps to reproduce, expected and actual results, severity, priority and the test environment. If a developer cannot reproduce the problem from the report alone, your developers will spend their time asking the vendor for missing details.
  • A test completion report. This is the summary written when a test level or project ends. Section 5.3.2 says it typically covers deviations from the test plan, unmitigated risks and defects not fixed. If a sample shows no deviations, no open risks and no unfixed defects, ask how the vendor established that.
  • Exit criteria they have used. Exit criteria are the conditions, agreed in advance, that must be met before testing counts as done. Ask what the vendor did when they reached a deadline before the criteria were met.
  • A list of subcontractors. Some vendors pass part of the work to other firms. The PRA said TSB relied on confirmations that SABIS's own suppliers gave to SABIS. It did this "without verifying whether SABIS had critically assessed these".

Contract terms that turn promises into deliverables

The contract is where each promise in the proposal becomes something you can check.

  • List the work products. Name the test cases, defect reports, progress reports and any automation code the vendor will deliver. State that your company owns them, so you keep the tests if you change vendors.
  • Name the key people. Name the test lead and the core testers. Require your approval before anyone on that core team is replaced.
  • Approve subcontractors. The vendor should not hand your work to another firm without your written consent.
  • Set the reporting rhythm. The syllabus says test progress reports go out on a regular schedule. They cover deviations, impediments and new risks. Put the frequency and contents in the contract.
  • Agree who decides "done". Write the exit criteria into the contract. Your team should decide when they are met, not the vendor.
  • Cover personal data. If the vendor will process personal data for you under the EU's General Data Protection Regulation (2016), Article 28 applies. It requires a written contract or another binding legal act. It also requires your prior written authorisation before they bring in another processor, and your right to audit. Where you can, give vendors masked test data, with personal details replaced, instead of production data.

Warning signs before you sign

A vendor may refuse a paid pilot on your system and offer only a demo on its own sample app. That vendor is asking you to trust the proposal. A vendor that reports only a pass rate gives you no way to judge what passed or why. Also watch for readiness statements in the future tense, such as "we expect to complete" or "we are confident". Ask what has been done, who did it, and where the evidence is kept. A team that changes between the pilot and the start of the contract means you are paying for people you have not met.

Your own team still has to read the vendor's reports and judge them, and Nopmark can talk with you about training your team to do that.

The vendor you choose will produce many of the results your release decision depends on. If you choose on promises, you will be judging those results without knowing how they were made. When you have seen the team work on your system and the contract lists what they deliver, you can check each result yourself.

Sources

Keep reading