Pairwise Testing Technique overview with implementation using PICT tool

Pairwise testing is a Combinatorial Testing technique also known as ‘All-Pairs Testing’.  While testing a system with multiple combinations, it is very hard to test all the combinations. Pairwise testing is a permutations and combinations based method, in which to test a system or an application, for each pair of input parameters of a system, all possible discrete combinations of the parameters are tested. This is a smart way of reducing the test combinations thus reducing the testing time without compromising the test coverage.

Why do we need it?

Lets take an example of Marketing email delivery API of a e-learning company. Consider that API takes Age, Gender, Geography and Last Purchased Course Category of user as parameters and decides the email content with Course suggestions etc. to the user.

Let’s list down the variables involved here:

1.  Age

     i)    10-20

     ii)   20-30

     iii)  30-40

     iv)  40-50

2.  Gender

     i)   Male

     ii)  Female

3.  Geography

     i)    EMEA

     ii)   NA

     iii)  LATAM

     iv)  APAC

4.  Last Purchased Course Category

     i)    Marketing.

    ii)    IT and Software.

    iii)   Personal Development.

    iv)   Business.

    v)    Music.

Now to test this API by using the conventional or exhaustive testing approach, we will have

 = 4*2*4*5 = 160 combinations.

In the current agile environment, it is hard for the testers to test all these combinations due to the tight timelines. Pairwise testing technique will help us significantly reducing these combinations in such a way that, we will have 100% coverage and high defect yield rate.

The process of generating these combinations manually through Pairwise testing technique involves placing each catogery in such a way that each combination is repeated once. There are lot of resources available to explain this process in internet. 

In this post, we will focus on generating these combinations automatically with the help of PICT tool.

we have few tools which can help us do this automatically. 

Tools available:

• PICT – ‘Pairwise Independent Combinatorial Testing’, provided by Microsoft Corp.
• IBM FoCuS – ‘Functional Coverage Unified Solution’, provided by IBM.
• ACTS – ‘Advanced Combinatorial Testing System’, provided by NIST, an agency of the US Government.
• Hexawise
• Jenny
• Pairwise by Inductive AS
• VPTag free All-Pair Testing Tool

Check here for more tools.

Using PICT for Pairwise Testing:

Among all the tools available, PICT is the widely used open source tool for generating combinations through Pairwise technique. But it is available only in windows.

Let’s start generating the combinations for the above example using PICT.

First we need to install it. You can download the PICT from here. After downloading this .exe file, just click on it, and installation steps are self-explanatory.

To confirm the installation is successful, open Command prompt and type ‘pict’. You should see below text:

The next step is, we need to create an input model in .txt file. PICT uses this model as an input for the combinations.

The input file will look like this.

We use colon(:) to separate parameters and their real values. All the parameter values are split by comma(,).

Now open your command prompt, navigate to the folder where PictExample.txt file is located using: cd path_to_PictExample.txt

Execute the following command to get the combinations: pict PictExample.txt

The result is:

We can redirect these combinations into excel sheet by using pict PictExample.txt > PictExample-combinations.xls

So, now you can see that, 160 combinations got reduced to 22 using Pairwise testing technique. You can also observe that we created maximum test coverage through these combinations.

Excluding the conflict combinations:

Consider that Music catogery is not available in NA region. But PICT generated that combination as well. We can eliminate that combination for future generations by excluding this combination in the input file itself like below:

Note: PICT is case sensitive about parameters and options. The exclusion condition should end with semicolon (;)

If you execute pict PictExample.txt command now, you will see the below output by removing the invalid combination.

Advantages:

• Pairwise testing reduces the number of test cases. So, less time to complete the execution thus saving the time.
• Pairwise testing increases the test coverage almost up to hundred percentage.
• Pairwise testing increases the defect detection ratio.

Disadvantages:

• Pairwise testing is not beneficial if the values of the variables are inappropriate.
• In pairwise testing it is possible to miss a highly probable combination while selecting the test data.
• In pairwise testing, defect yield ratio may be reduced if a combination is missed.
• Pairwise testing is not useful if the combination of variables are not understood correctly.