Exploring fault types, detection activities, and failure severity in an evolving safety-critical software system

Abstract

Many papers have been published on analysis and prediction of software faults and/or failures, but few established the links from software faults (i.e., the root causes) to (potential or observed) failures and addressed multiple attributes. This paper aims at filling this gap by studying types of faults that caused software failures, activities taking place when faults were detected or failures were reported, and the severity of failures. Furthermore, it explores the associations among these attributes and the trends within releases (i.e., pre-release and post-release) and across releases. The results are based on the data extracted from a safety-critical NASA mission, which follows an evolutionary development process. In particular, we analyzed 21 large-scale software components, which together constitute over 8,000 files and millions of lines of code. The main insights include: (1) only a few fault types were responsible for the majority of failures pre-release and post-release, and across releases. Analysis and testing activities detected the majority of failures caused by each fault type. (2) The distributions of fault types differed for pre-release and post-release failures. (3) The percentage of safety-critical failures was small overall, and their relative contribution increased on-orbit. (4) Both post-release failures and safety-critical failures were more heavily associated with coding faults than with any other type of faults. (5) Components that experienced high number of failures in one release were not necessarily among high failure-prone components in the subsequent release. (6) Components that experienced more failures pre-release were more likely to fail post-release, overall and for each release.

Appendix: Background on \(\chi ^2\) test for contingency tables

For any two attributes (i.e., random variables $X$ and $Y$ with $n$ and $m$ categories, respectively), we use the $\chi ^2$ test to explore if the distributions of failures across the $m$ categories of $Y$ are the same for each of the $n$ samples (i.e., categories) in $X$. That is, we test the null hypothesis which specifies no association between the two bases of classification. We build an $m$ x $n$ contingency table using the observed frequencies for each pair of categories $X_i$ and $Y_j$, and then calculate the standard $\chi ^2$ statistic which can be approximated by a chi-square distribution with $(m-1) (n-1)$ degrees of freedom. When used for contingency comparisons, the chi-square test is a nonparametric test, since it compares entire distributions rather than parameters of distributions.

Note that the $\chi ^2$ test is applicable to data in a contingency table only if the expected frequencies are sufficiently large. Specifically, for contingency tables with degrees of freedom >1, the $\chi ^2$ statistics may be used if no cell has an expected frequency <1 and fewer than 20 % of the cells have an expected frequency <5 (Cochran 1954). For rare categories that are not of interest, categories can be combined in order to increase the expected frequencies in various cells. However, rare categories that are considered important will necessitate larger sample size, both to be able to run the $\chi ^2$ test and to achieve adequate power (Kraemer and Thiemann 1987).

The power of a statistical test is the probability that the false null hypothesis will be rejected, that is, that the test will recognize an existing pattern in the data. Although an intuitive response may be to choose a power value around 99 % to be almost certain of demonstrating significance if the alternative hypothesis is true, that is usually impossible because the number of instances required per category is going to be prohibitive. Generally, statistical power in the 70–90 % range is considered acceptable (Kraemer and Thiemann 1987).

Once the $\chi ^2$ statistic has been calculated, we determine the probability under the null hypothesis of obtaining a value as large as the calculated $\chi ^2$ value. If the probability is equal to or less than the significance level $\alpha$ (in our case $\alpha =0.05$), the null hypothesis is rejected. Since the null hypothesis states that there is no relationship between the two variables (i.e., any correlation observed in the sample is due to chance), rejecting the null hypothesis implies that the distribution of failures across the $m$ categories differs significantly for the $n$ samples and suggests that there is some correlation between the two variables. Therefore, we also measure the extent of the correlation between the two variables. We use the contingency coefficient $C$ as a measure of correlation, since it is uniquely useful in cases when the information about at least one of the attributes is categorical (i.e., given on a nominal scale). The contingency coefficient $C$ does not require underlying continuity for the various categories used to measure either one or both attributes. Even more, it has the same value regardless of how the categories are arranged in the rows and columns. The contingency coefficient $C$ is calculated as:

$$\begin{aligned} C=\sqrt{\frac{\chi ^2}{N+\chi ^2}} \end{aligned}$$

(1)

where $N$ is the total number of observations and the value of $\chi ^2$ statistics is computed based on the contingency table (Seigel and Castellan 1988). Since the test of significance for the contingency coefficient $C$ is based solely on the $\chi ^2$ statistics, it follows that if the null hypothesis that the $n$ samples come from the same distribution is rejected, then the calculated $C$ value will be significant.

It is important to note that, unlike the other measures of correlation, the maximum value of $C$ is not equal to 1; it rather depends on the size of the table. Specifically, the maximum value of the contingency coefficient $C_{\rm max}$ is given by:

$$\begin{aligned} C_{\rm max}=\root 4 \of {\frac{m-1}{m} \cdot \frac{n-1}{n}} \end{aligned}$$

(2)

where $m$ is the number of rows and $n$ is the number of columns in the contingency table. Hence, even small values of $C$ often may be evidence of statistically significant correlation between variables. Further, $C$ values for contingency tables with different sizes are not directly comparable. However, by normalizing $C$ with the corresponding $C_{\rm max}$ as in equation (3), we ensure that the range will be between 0 and 1, and hence, $C^*$ values for different-sized tables can be compared (Blaikie 2003).

$$\begin{aligned} C^*=C/C_{\rm max}. \end{aligned}$$

(3)