The definition of quality metrics to estimate the mutation operator effectiveness can help us identify the most valuable mutation operators and, eventually, a subset of operators providing a similar performance than the whole set.

The Coverage-Based Quality Metric aims to value mutation operators depending on their usefulness to refine a test suite. The novelty of this quality metric is that it tackles the coverage of the test suite, as not all the mutants are executed by all the test cases. This metric works under the following two assumptions:

• Mutants killed by few of the test cases that reach them should be more valuable than those mutants killed by many of the test cases that reach them.
• Mutants killed by test cases that kill only few of the mutants reached by them should be more valuable than those mutants killed by test cases that kill many of the mutants reached by them.

The Coverage-Based Quality Metric addresses a special case: when there is a lack of coverage information for some of the mutants, we say that these mutants are difficult to reach. As we do not have enough coverage information about these mutants, it is an open question whether these mutants are easy or difficult to kill. We seek to categorise these mutants and favour them with the new quality metric because they might provide with necessary test cases.

Definition of difficult to reach mutants

Let MCOV be a constant representing the minimum number of coverage data required so that the coverage is regarded as significant. A dead mutant m is difficult to reach when the sum of the test cases reaching m and the non-equivalent mutants reached by those test cases is less or equal than MCOV.

As it can be seen from this definition, we need to set the value for MCOV. To this end, we carried out a sensitive analysis for our six C++ programs under study. We calculated the coverage-based quality metric of each class mutation operator applicable to each program using different thresholds for MCOV (from 2 to 6). The results for each case study are presented below:

• Case study 1 (Tcl)
• Case study 2 (Rpc)
• Case study 3 (Dph)
• Case study 4 (Kmy)
• Case study 5 (Txm)
• Case study 6 (Dom)

We can observe that, when using the values 2 and 3, several of the operators change their coverage-based quality metric. However, when using the values 5 and 6:

• There are scattered changes.
• These changes are often meaningful with respect to previous values for MCOV.

In the light of these results, we conclude that the value MCOV = 4 is an appropriate threshold to avoid overfitting the results to our programs under study.

The execution matrix (results of the execution of each mutant against each test case) and the coverage matrix (test cases that cover each mutant) associated with the programs under study can be downloaded from this link: