Unit testing performance with Stochastic Performance Logic

Abstract

Unit testing is an attractive quality management tool in the software development process, however, practical obstacles make it difficult to use unit tests for performance testing. We present Stochastic Performance Logic, a formalism for expressing performance requirements, together with interpretations that facilitate performance evaluation in the unit test context. The formalism and the interpretations are implemented in a performance testing framework and evaluated in multiple experiments, demonstrating the ability to identify performance differences in realistic unit test scenarios.

Appendix

1.1 Expected-value-based interpretation

The following lemma and theorem show that the interpretation of $=_p$ and $\le _p$, as defined by Definition 6, is consistent with axioms (4) and (5). The consistency with other axioms trivially results from the assumption of total ordering on W.

Lemma 1

Let $X, Y : \varOmega \rightarrow \mathbb {R}^+$ be random variables, and $tl, tr, tx, ty: \mathbb {R}^+ \rightarrow \mathbb {R}^+$ be performance observation transformation functions. Then the following holds:

$$\begin{aligned}&(\forall o \in \mathbb {R}^+: tl(o) \le tr(o)) \rightarrow E(tl(X)) \le E(tr(X))\\&\quad (E(tx(X)) \le E(ty(Y)) \wedge E(ty(Y)) \le E(tx(X))) \\&\quad \leftrightarrow E(tx(X)) = E(ty(Y)) \end{aligned}$$

Proof

The validity of the first formula follows from the definition of the expected value. Let f(x) be the probability density function of random variable X. Since $f(x) \ge 0$, it holds that

$$\begin{aligned} E(tl(X)) = \int ^\infty _{-\infty } tl(x) f(x) dx \le \int ^\infty _{-\infty } tr(x) f(x) dx = E(tr(X)) \end{aligned}$$

The validity of the second formula follows naturally from the properties of total ordering on real numbers.

Note that we assumed X and Y to be continuous random variables. The proof would be the same for discrete random variables, except with a sum in place of the integral.

Theorem 1

The interpretation of performance relations $\le _p$ and $=_p$, as given by Definition 6, is consistent with axioms (4) and (5).

Proof

The proof of the theorem naturally follows from Lemma 1 by substituting $P_M(x_1, \ldots , x_m)$ for X and $P_N(y_1, \ldots , y_n)$ for Y.

1.2 Quick mean-value interpretation

We need to show that the quick mean-value interpretation of SPL is consistent with axioms (4) and (5). Briefly, the interpretation relies on the Welch’s t test, which rejects with significance level $\alpha$ the null hypothesis $\overline{X} = \overline{Y}$ against the alternative hypothesis $\overline{X} \ne \overline{Y}$ if

$$\begin{aligned} \left| \frac{\overline{X} - \overline{Y}}{\sqrt{\frac{S_X^2}{V_X} + \frac{S_Y^2}{V_Y}}}\right| > t_{\nu , 1-\alpha /2} \end{aligned}$$

and rejects with significance level $\alpha$ the null hypothesis $\overline{X} \le \overline{Y}$ against the alternative hypothesis $\overline{X} > \overline{Y}$ if

$$\begin{aligned} \frac{\overline{X} - \overline{Y}}{\sqrt{\frac{S_X^2}{V_X} + \frac{S_Y^2}{V_Y}}} > t_{\nu , 1-\alpha } \end{aligned}$$

where $V_i$ is the sample size, $S_i^2$ is the sample variance, $t_{\nu , \alpha }$ is the $\alpha$-quantile of the Student’s distribution with $\nu$ levels of freedom, with $\nu$ computed as follows:

$$\begin{aligned} \nu = \frac{\left( \frac{S_X^2}{V_X} + \frac{S_Y^2}{V_Y}\right) ^2}{\frac{S_X^4}{V_X^2(V_X - 1)} + \frac{S_Y^4}{V_Y^2(V_Y - 1)}} \end{aligned}$$

Theorem 2

The interpretation of relations $=_p$ and $\le _p$, as given by Definition 8, is consistent with axiom (4) for a given fixed experiment $\mathcal {E}$.

Proof

For sake of brevity, we denote the sample mean of $tl(P_M^i(x_1, \ldots , x_m))$ as $\overline{M}_{tl}$ and the sample variance of the same as $S^2_{tl}$; $\overline{M}_{tr}$ and $S^2_{tr}$ are defined in a similar way.

Assuming $\forall o \in \mathbb {R}^+: tl(o) \le tr(o)$, we have to prove that the null-hypothesis $H_0: E(tl(P_M^i(x_1, \ldots , x_m))) \le E(tr(P_M^i(x_1, \ldots , x_m)))$ cannot be rejected by the Welch’s t test.

Based on the formulation of the t test, it means that the null-hypothesis can be rejected if

$$\begin{aligned} \frac{\overline{M}_{tl} - \overline{M}_{tr}}{\sqrt{\frac{S_{tl}^2}{V} + \frac{S_{tr}^2}{V}}} > t_{\nu , 1-\alpha } \end{aligned}$$

where V is the number of samples $P_M^i(x_1, \ldots , x_m)$ in the experiment $\mathcal {E}$.

Since the denominator is a positive number, the whole fraction is non-positive. However, the right hand side $t_{\nu , 1-\alpha }$ is a non-negative number since we assumed that $\alpha \le 0.5$. This means that the inequality never holds and thus the null-hypothesis cannot be rejected.

Theorem 3

The interpretation of relations $\le _p$, and $=_p$, as given by Definition 8, is consistent with axiom (5) for a given fixed experiment $\mathcal {E}$.

Proof

For sake of brevity, we denote the sample mean of $tm(P_M^i(x_1, \ldots , x_m))$ as $\overline{M}$ and the sample variance of the same as $S^2_{M}$, $\overline{N}$ and $S^2_{N}$ are similarly defined for N.

By interpreting axiom (5) according to Definition 6, we get the following statements:

$$\begin{aligned}&P_M(x_1, \ldots , x_m) \le _{p(tm, tn)} P_N(y_1, \ldots , y_n)\\&\quad \longleftrightarrow \frac{\overline{M} - \overline{N}}{\sqrt{\frac{S_M^2}{V_M} + \frac{S_N^2}{V_N}}} \le t_{\nu _{M,N}, 1-\alpha }\\&P_N(y_1, \ldots , y_n) \le _{p(tn, tm)} P_M(x_1, \ldots , x_m)\\&\quad \longleftrightarrow \frac{\overline{N} - \overline{M}}{\sqrt{\frac{S_N^2}{V_N} + \frac{S_M^2}{V_M}}} \le t_{\nu _{N,M}, 1-\alpha }\\&P_M(x_1, \ldots , x_m) =_{p(tm, tn)} P_N(y_1, \ldots , y_n)\\&\quad \longleftrightarrow -t_{\nu _{M,N}, 1-\alpha } \le \frac{\overline{M} - \overline{N}}{\sqrt{\frac{S_M^2}{V_M} + \frac{S_N^2}{V_N}}}\le t_{\nu _{M,N}, 1-\alpha } \end{aligned}$$

Thus, we need to show that

$$\begin{aligned}&\frac{\overline{M} - \overline{N}}{\sqrt{\frac{S_M^2}{V_M} + \frac{S_N^2}{V_N}}} \le t_{\nu _{M,N}, 1-\alpha } \wedge \frac{\overline{N} - \overline{M}}{\sqrt{\frac{S_N^2}{V_N} + \frac{S_M^2}{V_M}}} \le t_{\nu _{N,M}, 1-\alpha }\\&\quad \longleftrightarrow -t_{\nu _{M,N}, 1-\alpha } \le \frac{\overline{M} - \overline{N}}{\sqrt{\frac{S_M^2}{V_M} + \frac{S_N^2}{V_N}}}\le t_{\nu _{M,N}, 1-\alpha } \end{aligned}$$

This holds, because $\nu _{M,N} = \nu _{N,M}$ and thus

$$\begin{aligned} \frac{\overline{N} - \overline{M}}{\sqrt{\frac{S_N^2}{V_N} + \frac{S_M^2}{V_M}}} \le t_{\nu _{N,M}, 1-\alpha } \longleftrightarrow -t_{\nu _{X,Y}, 1-\alpha } \le \frac{\overline{M} - \overline{N}}{\sqrt{\frac{S_M^2}{V_M} + \frac{S_N^2}{V_N}}} \end{aligned}$$

Note that, as indicated in Sect. 2, transitivity (i.e., $(P_X(\ldots ) \le _{p(tx,ty)} P_Y(\ldots ) \wedge P_Y(\ldots ) \le _{p(ty,tz)} P_Z(\ldots )) \rightarrow P_X(\ldots ) \le _{p(tx,tz)} P_Z(\ldots )$) does not hold for the sample-based interpretation. This can be shown by considering the following observations and performing single-sided tests at significance level $\alpha =0.05$: $\mathcal {O}_{P_X}=\{2,4\}$, $\mathcal {O}_{P_Y}=\{-1,1\}$, $\mathcal {O}_{P_Z}=\{-4,-2\}$.

1.3 Parametric mean-value interpretation

We need to show that the parametric mean-value interpretation of SPL is consistent with axioms (4) and (5).

Theorem 4

The interpretation of relations $\le _p$, and $=_p$, as given by Definition 9, is consistent with axioms (4) and (5) for a given fixed experiment $\mathcal {E}$.

Proof

The proof is very similar to the proofs of Theorems 2 and 3. To prove consistency with axiom (4), it is necessary to show that

$$\begin{aligned} \overline{M_{tl}}-\overline{M_{tr}} \le z_{(1-\alpha )}\sqrt{\frac{o_MR_{tl}^2+\overline{S^2_{tl}}}{r_Mo_M}+\frac{o_MR_{tr}^2+\overline{S^2_{tr}}}{r_Mo_M}} \end{aligned}$$

where $M_{tl}$ denotes the sample mean of run means of $tl(P^{i,j}_M(x_1, \ldots , x_m))$, $\overline{S^2_{tl}}$ and $R^2_{tl}$ denote for the same the average of variances of samples per run and the variance of run means, respectively. $M_{tr}$, $\overline{S^2_{tr}}$, and $R^2_{tr}$ are similarly defined for $tr(P^{i,j}_M(x_1, \ldots , x_m))$. This inequality is obviously true because $\overline{M_{tl}}\le \overline{M_{tr}}$ and because the right-hand side of the inequality is non-negative.

The consistency with axiom (5) directly comes from comparing the first part of Definition 9 with the second part of the definition.

1.4 Non-parametric mean-value interpretation

To show that the non-parametric mean-value interpretation of SPL is consistent with axioms (4) and (5), we assume pure bootstrap without the Monte-Carlo simulations. We further restrict tl, tr to be linear functions and assume $\alpha \le 0.35$ as reasonable practical restrictions.

Theorem 5

Assuming that tl and tr are linear functions, the interpretation of relations $\le _p$, and $=_p$, as given by Definition 10, with M and N used for X and Y, is consistent with axiom (4) for a given fixed experiment ${\mathcal {E}}$ and $\alpha \le 0.35$.

Proof

To prove the consistency of $\le _p$ with axiom (4), we have to show that the following condition holds

$$\begin{aligned} B^*_{\overline{X}_{tl}, r, o - \overline{X}_{tr}, r, o}(\overline{X}_{tl}-\overline{X}_{tr}) \le 1-\alpha \end{aligned}$$

(6)

The left-hand side of the expression above, i.e., $B^*_{\overline{X}_{tl}, r, o - \overline{X}_{tr}, r, o}(\overline{X}_{tl}-\overline{X}_{tr})$, corresponds to

$$\begin{aligned}&\frac{1}{|\mathcal {S}|^{2}}\!\sum _{s_{tl}\in \mathcal {S}}\sum _{s_{tr}\in \mathcal {S}}\mathbf {1}\left( \frac{1}{ro}\sum _{i=1}^{r} \sum _{j=1}^{o}\left( tl\left( P_X^{s_{tl}(i,j)}\right) - tr\left( P_X^{s_{tr}(i,j)}\right) \right) \right. \\&\quad \left. - (\overline{X}_{tl} - \overline{X}_{tr}) \le \overline{X}_{tl} - \overline{X}_{tr}\right) \end{aligned}$$

where $\mathbf {1}$ is an indicator function; and $s_{tl}, s_{tr}: \mathbb {N}\times \mathbb {N}\rightarrow \mathbb {N}\times \mathbb {N}$ are selector functions that determine observations to be drawn with replacement from observations $P^{i,j}_X(x_1, \ldots , x_m)$. We denote $\mathcal {S}$ as the set of all selectors for the observations.

Since $tl(x)=a_{tl}x + b_{tl}$, $tr(x)=a_{tr}x + b_{tr}$ are linear functions, we can rewrite the condition (6) as

$$\begin{aligned} \frac{1}{|\mathcal {Z}|^{2}}\sum _{i=1}^{|\mathcal {Z}|}\sum _{j=1}^{|\mathcal {Z}|} \mathbf {1}\left( a_{tl}z_i-a_{tr}z_j + b_{tl} - b_{tr} \le 2(a_{tl} - a_{tr})\overline{X} + 2(b_{tl} - b_{tr})\right) \le 1-\alpha \end{aligned}$$

where Z is a multiset defined as

$$\begin{aligned} \mathcal {Z}=\left\{ \frac{1}{ro}\sum _{i=1}^{r}\sum _{j=1}^{o} P_X^{s(i,j)} \Bigg | \forall s\in \mathcal {S}\right\} \end{aligned}$$

and $z_i \in \mathcal {Z}$ is a sorted sequence (i.e., $z_1 \le \cdots \le z_{|\mathcal {Z}|}$) consisting of all items in $\mathcal {Z}$.

To prove (6), let us assume that it does not hold. Then the $1-\alpha$ quantile (denoted $q_{1-\alpha }$) of $B^*_{\overline{X}_{tl}, r, o - \overline{X}_{tr}, r, o}$ would have to be less than $2(a_{tl} - a_{tr})\overline{X} + 2(b_{tl} - b_{tr})$. We show that this situation cannot happen for $\alpha = 0.35$ and consequently for any $\alpha \le 0.35$ (because $q_{1-0.35} \le q_{1-\alpha }$ for $\alpha \le 0.35$). For the sake of contradiction, we posit

$$\begin{aligned} q_{1-0.35} < 2(a_{tl} - a_{tr})\overline{X} + 2(b_{tl} - b_{tr}) \end{aligned}$$

(7)

Fig. 13

Partial ordering of differences $a_{tl}z_i - a_{tr}z_j$

Rewriting the sample mean $\overline{X}$ as an average of values $z_i$, we get

$$\begin{aligned} q_{1-0.35} < 2(a_{tl}-a_{tr})\frac{\sum _{i=1}^{|\mathcal {Z}|}z_i}{|\mathcal {Z}|} + 2(b_{tl} - b_{tr}) \end{aligned}$$

(8)

Next, we approximate the quantile $q_{1-0.35}$ with $(a_{tl}-a_{tr})z_{\lceil (|\mathcal {Z}|+1)/2\rceil }$

$$\begin{aligned} (a_{tl}-a_{tr})z_{\left\lceil \frac{|\mathcal {Z}|+1}{2}\right\rceil } + b_{tl} - b_{tr} \le q_{1-0.35} \end{aligned}$$

This holds because the partial ordering of the differences $a_{tl}z_i - a_{tr}z_j$ forms a lattice as shown in Fig. 13 (the arrows point from smaller values to larger ones). Thanks to this partial order, at least 35  of the differences $a_{tl}z_i - a_{tr}z_j$ are not smaller than $(a_{tl}-a_{tr})z_{\lceil (|\mathcal {Z}|+1)/2\rceil }$ (the values delineated in the figure by the dashed and dotted red line, which is formed by a rectangular area of 1 / 4 of all values above the center and a triangular area of 1 / 8 of all values left of the center). Consequently $(a_{tl}-a_{tr})z_{\lceil (|\mathcal {Z}|+1)/2\rceil } + b_{tr} - b_{tl}$ has to be less or equal to $q_{1-0.35}$. We thus rewrite (8) as

$$\begin{aligned} (a_{tl}-a_{tr})z_{\left\lceil \frac{|\mathcal {Z}|+1}{2}\right\rceil } + b_{tl} - b_{tr} \le q_{1-0.35} < 2(a_{tl}-a_{tr})\frac{\sum _{i=1}^{|\mathcal {Z}|}z_i}{|\mathcal {Z}|} + 2(b_{tl} - b_{tr})\nonumber \\ \end{aligned}$$

(9)

Since $tr(o) \ge tl(o)$, we have $a_{tl}-a_{tr} \le 0$ and $b_{tl}-b_{tr} \le 0$. After removing $b_{tl}-b_{tr}$, multiplying the inequality by $|Z|/(a_{tl}-a_{tr})$ and approximating the sum on the right hand side of (9) we get the expected contradiction, which completes the proof for $\le _p$.

$$\begin{aligned} |\mathcal {Z}|z_{\left\lceil \frac{|\mathcal {Z}|+1}{2}\right\rceil } > 2\sum _{i=1}^{|\mathcal {Z}|}z_i \ge 2\left\lceil \frac{|\mathcal {Z}|+1}{2}\right\rceil z_{\left\lceil \frac{|\mathcal {Z}|+1}{2}\right\rceil } \ge \left( |\mathcal {Z}|+1\right) z_{\left\lceil \frac{|\mathcal {Z}|+1}{2}\right\rceil } \end{aligned}$$

Note that $(a_{tl}-a_{tr})$ must be negative and $z_i$ is positive because it averages performance observations, which are positive themselves. For the case $a_{tl} = a_{tr}$, the contradiction in (7) is already clear from (9).

The proof for $=_p$ is similar. It has to additionally show that

$$\begin{aligned} \alpha \le B^*_{\overline{X}_{tl}, r, o - \overline{X}_{tr}, r, o}(\overline{X}_{tl}-\overline{X}_{tr}) \end{aligned}$$

which can be done following the same reasoning as already shown in the proof above. $\square$

Theorem 6

The interpretation of relations $\le _p$, and $=_p$, as given by Definition 10, is consistent with axiom (5) for a given fixed experiment $\mathcal {E}$.

Proof

Similar to the proof of consistency with axiom (5) in Theorem 4, the consistency here also directly comes from comparing the first part of Definition 10 with the second part of the definition.