Customer-Perceived Software Reliability Predictions: Beyond Defect Prediction Models

Abstract

In this chapter, we propose a procedure for implementing customer-perceived software reliability predictions, which address customer’s concern about service-impacting outages and system stability. Data requirements are clearly defined in terms of test defects and field outages to ensure a good data collection process. We incorporate the effect of operational profile to demonstrate the changes in defect find rate from internal tests through precutover test and in-service operation. A software reliability growth model is a necessary key step, but not sufficient for addressing customer-perceived reliability measures. The proposed approach is a result of in-depth investigations of test defect data and field outage data over many years. It has been successfully demonstrated with actual field data and applied to a variety of software development projects.

Appendix A: Derivation of Maximum Likelihood Estimates for an Exponential Model

In this section, we will consider a case where defect data are available on a grouped basis such as weekly. The equations for deriving estimates of a and b for an exponential model will be provided using the maximum likelihood estimation method. Additional details are available from Musa et al. [6].

Let $y_{i}(i = 1, \ldots , p)$ be the number of defects found in $(0, x_{i})$. Then the likelihood function of a and b, given the defect data set $y_{i}(i = 1, \ldots , p)$, is derived from (1) as:

$$\begin{aligned} L(\mathbf{a }, \mathbf{b }; y_{1}, \ldots , y_{p}) = \prod _{i=1}^{p} \text{ m }( x_{i} - x_{i-1})^{y_{i} - y_{i-1}} {\text{ exp }} \{-\text{ m }(x_{i}-x_{i-1})\} / (y_{i}- y_{i-1})! \end{aligned}$$

(A.1)

where $x_{0 }=y_{0}= 0$, and, the mean value function m(t) is given by (2). After some algebra by taking partial derivatives of the log-likelihood function of (A.1) with respect to a and b and setting to zeros, we have the following two equations:

$$\begin{aligned} \mathbf{a } = y_{p} / [1 - {\text{ exp }}(- \mathbf{b }x_{p})] \end{aligned}$$

(A.2)

and

$$\begin{aligned} \sum \limits _{i=1}^{p} (\text{ A }_{i} / \text{ B }_{i}) - \text{ C } = 0 \end{aligned}$$

(A.3)

where $\text{ A }_{i}, \text{ B }_{i}$, and C are, respectively, given by:

$$\begin{aligned} \text{ A }_{i} = (y_{i}- y_{i-1}) [x_{i} {\text{ exp }}(-\mathbf{b } x_{i}) - x_{i-1} {\text{ exp }}(-\mathbf{b } x_{i-1})] \end{aligned}$$

(A.4)

$$\begin{aligned} \text{ B }_{i} = {\text{ exp }}(-\mathbf{b } x_{i-1}) - {\text{ exp }}(-\mathbf{b }x_{i}) \end{aligned}$$

(A.5)

$$\begin{aligned} \text{ C } = x_{p} y_{p} / [{\text{ exp }}(\mathbf{b } x_{p}) - 1]. \end{aligned}$$

(A.6)

Maximum likelihood estimates of a and b can be obtained by solving Eqs. (A.2) and (A.3). Note that Eq. (A.2) implies a and b satisfy the last data point $(x_{p}, y_{p})$. That is, the mean value function with the maximum likelihood estimates of a and b always goes through the first data point $(x_{0 }, y_{0})$ and last data points point $(x_{p}, y_{p})$. It should be pointed out that Eq. (A.3) is nonlinear but can be easily implemented in a spreadsheet with the use of a built-in function such as “solver”.

In order to obtain confidence intervals for a and b, we take a second derivative with respect to b and substitute the estimate of b into the negative of the second derivative. Since the inverse of the above quantity is considered as the variance of estimate b, the 90 % confidence interval for b can be constructed using a normal approximation. The 90 % confidence interval for a can be obtained using (A.2) for each limit of b.