Does choice of mutation tool matter?

Abstract

Though mutation analysis is the primary means of evaluating the quality of test suites, it suffers from inadequate standardization. Mutation analysis tools vary based on language, when mutants are generated (phase of compilation), and target audience. Mutation tools rarely implement the complete set of operators proposed in the literature and mostly implement at least a few domain-specific mutation operators. Thus different tools may not always agree on the mutant kills of a test suite. Few criteria exist to guide a practitioner in choosing the right tool for either evaluating effectiveness of a test suite or for comparing different testing techniques. We investigate an ensemble of measures for evaluating efficacy of mutants produced by different tools. These include the traditional difficulty of detection, strength of minimal sets, and the diversity of mutants, as well as the information carried by the mutants produced. We find that mutation tools rarely agree. The disagreement between scores can be large, and the variation due to characteristics of the project—even after accounting for difference due to test suites—is a significant factor. However, the mean difference between tools is very small, indicating that no single tool consistently skews mutation scores high or low for all projects. These results suggest that experiments yielding small differences in mutation score, especially using a single tool, or a small number of projects may not be reliable. There is a clear need for greater standardization of mutation analysis. We propose one approach for such a standardization.

Appendix

1.1 Measures of correlation

We rely on two different correlations here: The first is $R^2$, which suggests how close variables are linearly. $R^2$ (Pearson’s correlation coefficient) is a statistical measure of the goodness of fit, that is, the amount of variation in one variable that is explained by the variation in the other. For our purposes, it is the ability of mutation scores produced by one tool to predict the score of the other. We expect $R^2 = 1$ if either A) scores given by both tools for same program are the same, or B) they are always separated by same amount. The Kendall’s $\tau _b$ is a measure of monotonicity between variables compared, measuring the difference between concordant and discordant pairs. Kendall’s $\tau _b$ rank correlation coefficient is a nonparametric measure of association between two variables. It requires only that the dependent and independent variables (here mutation scores from two different tools) are connected by a monotonic function. It is defined as

$$\begin{aligned} \tau _b = \frac{{\text {concordant\,pairs}} - {\text {discordant\,pairs}}}{\frac{1}{2}n(n-1)} \end{aligned}$$

$R^2$ and $\tau _b$ provide information along two different dimensions of comparison. That is, $R^2$ can be close to 1 if the scores from both tools are different by a small amount, even if there is no consistency in which one has the larger score. However, such data would result in very low $\tau _b$, since the difference between concordant and discordant pairs would be small. On the other hand, say the mutation scores of one tool are linearly proportional to the test suite, while another tool has a different relation to the test suite—say squared increase. In such a case, the $R^2$ would be low since the relation between the two tools is not linear, while $\tau _b$ would be high. Hence both measures provide useful comparative information. Note that low $\tau _b$ indicates that the troubling situation in which tools would rank two test suites in opposite order of effectiveness is more frequent—this could lead to a change in the results of software testing experiments using mutation analysis to evaluate techniques, just by changing the tool used for measurement.

While what can be considered high and low correlation is subjective, for the purpose of this paper, we consider $R^2 \le 0.40$ to be low correlation, and $R^2 \ge 0.60$ to be high correlation.

1.2 Covariance

We showed (Gopinath et al. 2015) that for mutation analysis, the maximum number of mutants to be sampled for given tolerance has an upper bound provided by the binomial distribution, and the actual number is determined by the covariance.

Let the random variable $D_n$ denote the number of detected mutants out of our sample n. The estimated mutation score is given by $M_n = \frac{D_n}{n}$. The random variable $D_n$ can be modeled as the sum of all random variables representing mutants $X_{1 \ldots n}$. That is, $D_n = \sum _{i}^{n} X_i$. The expected value of $E(M_n)$ is given by $\frac{1}{n}E(D_n)$. The variance $V(M_n)$ is given by $\frac{1}{n^2}V(D_n)$, which can be written in terms of component random variables $X_{1 \ldots n}$ as:

$$\begin{aligned} \frac{1}{n^2}V(D_n) = \frac{1}{n^2} \sum _{i}^{n} V(X_i) + 2\sum _{i<j}^{n}Cov(X_i,X_j) \end{aligned}$$

Using a simplifying assumption that mutants are more similar to each other than dissimilar, we can assume that

$$\begin{aligned} 2\sum _{i<j}^{n}Cov(X_i,X_j) >= 0 \end{aligned}$$

The sum of covariance will be zero when the mutants are independent. That is, the variance of the mutants $V(M_n)$ is strictly greater than or equal to that of a similar distribution of independent random variables.

This means that the covariance between mutants determines the size of the sample required. That is, the larger the covariance (or correlation) between mutants, the smaller the diversity.

1.3 Mutual information

The mutual information of a variable is defined as the reduction in uncertainty of a variable due to knowledge of another. That is, given two variables X and Y, the redundancy between them is estimated as:

$$\begin{aligned} I(X;Y) = I(Y;X) = \sum _{y\in Y} \sum _{x\in X} p(x,y) \log \bigg (\frac{p(x,y)}{p(x)p(y)} \bigg ) \end{aligned}$$

To extend this to a set of mutants, we use one of the multivariate generalizations of mutual information proposed by Watanabe (1960)—multi-information also called total correlation. The important aspects of multi-information that are relevant to us are that it is well behaved—that is it allows only positive values, and is zero only when all variables are completely independent. The multi-information for a set of random variables $x_i \in X$ is defined formally as:

$$\begin{aligned} C(X_1 \ldots X_n) = \sum _{x_1\in X_1} \ldots \sum _{x_n\in X_n} p(x_1 \ldots x_n)\log \bigg (\frac{p(x_1 \ldots x_n)}{p(x_1) \ldots p(x_n)} \bigg ). \end{aligned}$$

1.4 Entropy

In information theory, Shannon entropy (Shannon 2001) is a measure of the information content in the given data. Entropy is related to multi-information. That is, multi-information is the difference between the sum of independent entropies of random variables and their joint entropy. Formally,

$$\begin{aligned} C(X_1 \ldots X_n) = \sum _{i=1}^{N} H(X_i) - H(X_1 \ldots X_n) \end{aligned}$$

Another reason we are interested in the entropy of a set of mutants is that the properties of entropy are also relevant to how good we judge a set of mutants to be. That is, as we expect from a measure of quality of a set of mutants, the value can never be negative (adding a mutant to a set of mutants should not decrease the utility of a mutant set). Secondly, a mutant set where all mutants are killed by all test cases has minimal value (think of a minimal set of mutants for such a matrix). This is mirrored by the entropy property that $I(1) = 0$. Similarly, a mutant set where no mutants are killed by any test cases is also of no value (again consider the minimal set of mutants for such a matrix), which is also mirrored by entropy $I(0) = 0$. Finally, we expect that if two mutant sets representing independent failures are combined, the measure should reflect the sum of their utilities. With entropy, the joint information of two independent random variables is their sum of respective information. Finally, the maximum entropy for a set of mutants happens when none of the mutants in the set are subsumed by any other mutants in the set. The entropy of a random variable is given by:

$$\begin{aligned} I(p) = - p\times {}\log _2(p). \end{aligned}$$