How To Perform Pairwise Testing as a Test Analyst

Title 'How To Perform Pairwise Testing' on a white brush stroke over a blurred multiple-choice answer sheet with filled-in bubbles and a pencil.

At the beginning of ones testing career, testing may appear to be simple: you provide a software input and verify the outcome; if the output isn't what you expected, you file a bug report. However, software is rarely this straightforward; there are a wide range of input and output values, as well as unknown interdependencies between distinct modules. The outputs for the majority of test cases in a typical application may not be a single parameter, but rather a combination of elements such as state transitions, input parameters, ambient conditions, and so on. We may need to test across browsers, platforms, and devices, among other things. Software testers who have progressed far enough in their careers in software testing and want to expand their knowledge of test analysis and design start looking at formal certifications such as ISTQB Advanced Level Test Analyst.

One of those tools available to aid a Test Analyst is pairwise testing, a test case design approach that combines two separate factors to generate multiple permutations and combinations for optimum coverage.

What is Pairwise Testing?

Testing accounts for half of the expenditure of a typical commercial software project. Testing, on the other hand, isn't only costly; it's also arbitrarily expensive. This is due to the fact that there are more unique possible test cases for even a basic software product than can be completed within one's lifetime. As a result, pragmatic software testing necessitates implementing risk-based or assumptions-based optimizations. Each optimization approach has its own set of drawbacks.

Combinations are hard to test. When we have a product that processes numerous variables that may interact, we confront the difficulty of combinatorial testing. Variables can come from a variety of places, including the user interface, operating system, peripherals, a database, or a network. Combinatorial testing entails more than just evaluating individual variables (though that is necessary). The goal of combinatorial testing is to ensure that the system handles various combinations of variables appropriately.

An example of a combinatorial testing problem is the Settings dialog of Microsoft Word. Combinatorial testing is difficult because of the large number of possible test cases. For example, in the Microsoft Word settings dialog box below, only accounting for the checkboxes, there are over 4 million combinations (>22 different checkboxes). And since a defect may only result in a failure when a certain set of combinations of those checkboxes are checked/not-checked we may need to test them all depending on the risk of failure.

Screenshot of Microsoft Word's 'View' settings: 23 checkboxes under 'Show in Document', 'Show Non-Printing Characters', 'Show Window Elements' and 'Ribbon'.
When testing software, pairwise testing is used when various input parameters, each with several possible values, must be checked in conjunction, resulting in more combinations than can be examined in the time allocated to the testing effort. Pairwise testing is a useful test case creation approach based on the assumption that most failures are generated by interactions between at most two components. Pairwise-generated test suites cover all possible combinations of two, so they're significantly smaller than exhaustive ones, but they're still quite effective at detecting flaws.

Pairwise testing generates a collection of test cases for each pair of variables that covers all possible combinations of the test data values. All-pairs testing and 2-way testing are other names for pairwise testing. Of course, you can test all triples (3-way) or all quadruples (4-way), but the size of the higher order test sets grows quickly.

Choosing values for the software's input variables is usually the first step in pairwise testing. Domain partitioning is frequently used to pick these specific values. The values are then permuted to ensure that all of the pairings are covered. It can a long time to do this by hand. Many free, open source tools to produce pairwise test cases exist.

Let us consider a simple example of pairwise testing. Consider a software application that has three inputs X, Y, and Z. If each input variable can have only 2 values, then the total number of possible test cases is 2×2×2 = 8 test cases. Let's assume the set of values for X, Y, and Z are given below:

(X) = {1, 2}

(Y) = {YES, NO}

(Z) = {A, B}

A table of all possible combinations would be:

Table with columns Input X, Input Y and Input Z listing all 8 combinations of X (1 or 2), Y (YES or NO) and Z (A or B).

In the following table, we use the pairwise test set of size 4 is a 50% reduction from the full combinatorial test set of size 8. You can see from the table that every pair of values is represented in at least one of the rows.

Table with columns Input X, Input Y and Input Z showing 4 pairwise test rows: 1, YES, A; 1, NO, B; 2, YES, B; 2, NO, A.

If the number of variables and values per variable were to grow, the reduction in size of the test set would be more pronounced. For example, a full combinatorial test set of size 1029 can be reduced to 28 test cases.

Pairwise testing technique is just one of the techniques in the toolbox of an advanced tester. Other techniques include Boundary Value Analysis, Decision Table Testing, State Transition Testing, Classification Tree Technique, Use Case Testing, and much more.

The International Software Testing Qualifications Board (ISTQB) recognizes the skills an Advanced Level tester needs to possess, and have created a certification that demonstrates a software tester has these skills. The Advanced Level Test Analyst qualification is aimed at people who have achieved an advanced point in their careers in software testing. This includes people in roles such as testers, test analysts, test engineers, test consultants, test managers, user acceptance testers and software developers.

NOPMARK now offers training targeted to the Tester or Test Lead who is looking at taking their career to the next level and becoming a Test Analyst. Use promo code CTALTA10 for an introductory 10% discount on the new ISTQB Advanced Level Test Analyst online training course.

Keep reading