Mutant Accuracy Testing for Assessing the Implementation of Numerical Algorithms

Abstract

Despite their widespread use, implementations of numerical computing algorithms are generally tested manually with fuzzily defined thresholds determining success or failure. Modern software testing methods, such as automated regression testing, are difficult to apply because both test oracles and algorithm output are approximate. Based on the observation that high accuracy numerical algorithms appear to be fragile by design to errors in their parameters, we propose to compare the error of target implementations to mutated versions of themselves with the expectation that the mutants will suffer degraded accuracy. We test the idea on Matlab implementations of some basic numerical algorithms, and find that most mutants are worse while the few which are better show a distinctive pattern of mutation.

This work was supported by National Science and Engineering Research Council of Canada (NSERC) Discovery Grant RGPIN-2017-04543 and an Undergraduate Student Research Award (USRA).

A Numerical Algorithms

Our implementations are based on the descriptions in [1]. In this section we briefly describe each of the algorithms, the error measurement for that algorithm, and the test case generation procedure.

1.1 A.1 Simpson’s Method

The definite integral problem is to evaluate

$$ I = \int _a^b f(x)\,dx $$

for a specified $a,b \in \mathbb {R}$ and scalar function f(x).

Simpson’s method is a fourth order accurate method that is used to approximate definite integrals. It is given by:

$$I_{Simp} = \frac{b-a}{6}\left[ f(a) + 4f\left( \frac{b+a}{2}\right) + f(b)\right] $$

Composite Simpson’s method involves dividing the domain of integration into subintervals called “panels,” applying Simpson’s method to each panel, and summing the result. Let r, the number of panels, be even. The formula is

$$S_{comp}=\frac{h}{3}\left[ f(a) + 2\sum _{k=1}^{r/2-1} f(t_{2k}) + 4 \sum _{k=1}^{r/2} f(t_{2k-1}) + f(b)\right] $$

where $t_i = a+ih$, $i = \{1, 2, ..., r\}$.

For analysis purposes, we define the error as the absolute value of the difference between the algorithm’s output and the (floating point approximation of the) analytic answer. It can be shown that composite Simpson’s method has an error bound of

$$\frac{\Vert f^{(4)}\Vert _\infty }{180}(b-a)h^4.$$

A typical 3-line implementation of composite Simpson’s method in Matlab generates 180–190 mutants using MATmute, of which roughly 120–130 are viable.

Test cases are generated either using pen-and-paper integration, or by MMS.

1.2 A.2 Complete Cubic Spline

A cubic spline is a continuously differentiable, piecewise cubic scalar function v(x) that interpolates points $\{(x_1,f(x_1)), ..., (x_n,f(x_n))\}$, meaning that $v(x_i) = f(x_i)$. We call the spline “complete” because the derivatives at the endpoints $f'(x_1)$ and $f'(x_n)$ are also provided, and we choose v(x) such that $v'(x_1) = f'(x_1)$ and $v'(x_n) = f'(x_n)$.

For analysis purposes, we generate a set of test points between the interpolation points, compute the difference between the value of the interpolant and the value of the original function at these test points, and report the greatest absolute difference across all test points as the error.

MATmute generates roughly 680–700 mutants from our 30-line implementation of a complete cubic spline.

For this problem analytic solutions are trivial to construct: We pick a function f and a set of points $\{x_i\}$. Then, we use $\{x_i\}$ and $\{f(x_i)\}$ as the input to our cubic spline algorithm. The exact answers can be obtained directly from the function f.

1.3 A.3 Runge-Kutta Schemes

Runge-Kutta methods are used to solve initial value problems for ordinary differential equations. This class of problems is defined by:

$$\begin{aligned} \frac{dx(t)}{dt} = f(x, t) \text { such that } x(t_0) = x_0, \end{aligned}$$

where f(x) and $x_0$ are specified. While x(t) may be a vector in general, for our purposes we considered only scalar cases.

We used the classic fourth order accurate Runge-Kutta method. It is given by the following set of update formulae:

$$ \begin{aligned} y_{n+1}&= y_n + \frac{1}{6}(k_1 + k_2 + k_3 + k_4), \\ k_1&= hf(t_n,y_n), \\ k_2&= hf(t_n+h/2,y_n+k_1/2), \\ k_3&= hf(t_n+h/2,y_n+k_2/2), \\ k_4&= hf(t_n+h,y_n+k_3), \end{aligned} $$

where $y_n$ is the approximation at time $t_n$ and a fixed stepsize h has been assumed.

For analysis purposes, we define the error in our Runge-Kutta implementation as the absolute value of the difference between the algorithm’s output and the analytic answer at a final time.

MATmute generates roughly 480–500 mutants from a typical 4th order Runge-Kutta implementation.

Test cases are generated by MMS: Starting with a function for x(t), supply

$$ f(x,t) = \frac{dx(t)}{dt} \text { and } x_0 = x(t_0) $$

as the inputs.