The internal consistency and precedence of key process areas in the capability maturity model for software

Abstract

Evaluating the reliability of maturity level (ML) ratings is crucial for providing confidence in the results of software process assessments. This study investigates the dimensions underlying the maturity construct in the Capability Maturity Model (CMM) for Software (SW-CMM) and estimates the internal consistency of each dimension. The results suggest that SW-CMM maturity is a three-dimensional construct, with “Project Implementation” representing the ML 2 key process areas (KPAs), “Organization Implementation” representing the ML 3 KPAs, and “Quantitative Process Implementation” representing the KPAs at MLs 4 and 5. The internal consistency for each of the three dimensions as estimated by Cronbach’s alpha exceeds the recommended value of 0.9. Based on those results, this study builds and tests a theoretical model which posits that the achievement of lower ML KPAs sustains the implementation of higher ML KPAs. Results of path analysis using partial least squares (PLS) support the theoretical model and provide detailed understanding of the process improvement path. The analysis is based on 676 CMM-Based Appraisal for Internal Process Improvement (CBA IPI) assessments.

Appendices

Appendix 1

1.1 Estimating Cronbach’s Alpha

In discussing the reliability of measurements, a set of items (indicators) is posited to reflect an underlying construct. In the SW-CMM, maturity that is neither directly measurable nor observable can be indirectly measured by considering the assessed values of the KPAs. We can say that the SW-CMM uses an 18-item (or KPA) instrument to measure the process maturity of organizations. If the necessary data were readily available, we also could use a 52-item (goal) instrument (the SW-CMM includes 52 goals, with 20, 17, 6, and 9 goals in MLs 2, 3, 4, and 5, respectively).

The type of scale used in most assessment instruments is a summative one (Spector 1992). A summative scale means that the individual ratings x _i ’s for item i are summed up to produce an overall rating score (i.e., $y = {\sum\limits_{i = 1}^n {x_{i} } }$, where n denotes the number of items). One property of the covariance matrix for a summative rating is that the sum of all terms in the matrix gives exactly the variance of the scale as a whole, i.e., $\sigma ^{2}_{y} = {\sum\limits_{{\left( {i,j} \right)}} {\sigma _{{ij}} } }$, where i and j are indices of x _i and x _j ; if i = j, then $\sigma _{{ij}} = \sigma ^{e}_{i} ,$, i.e., variance of the ith item’s ratings (unique variation). The variability in a set of items score is considered to consist of the following two components:

• The error terms are the source of unique variation that each item possesses (i.e., ${\sum\limits_i {\sigma ^{2}_{i} } }$).

• The signal component of variance (covariance part) that is considered to be attributable to a common source due to the fact that maturity is the difference between total variance ${\left( {\sigma ^{2}_{y} } \right)}$ and unique variance ${\left( {{\sum\limits_i {\sigma ^{2}_{i} } }} \right)}$ (i.e., $\sigma ^{2}_{y} - {\sum\limits_i {\sigma ^{2}_{i} } }$).

The ratio of true to observed variance is ${\left( {\sigma ^{2}_{y} - {\sum\limits_i {\sigma ^{2}_{i} } }} \right)} \div \sigma ^{2}_{y}$. To express the ratio in relative terms, the number of elements in the covariance matrix of a summative rating must be considered. The total number of elements in the covariance matrix is n ², and the total number of communal elements is n ² − n. Thus, Cronbach’s alpha becomes the following:

$$ \alpha = \frac{n} {{{\left( {n - 1} \right)}}}{\left[ {1 - {\sum\limits_i {{\sigma ^{2}_{i} } \mathord{\left/ {\vphantom {{\sigma ^{2}_{i} } {\sigma ^{2}_{y} }}} \right. \kern-\nulldelimiterspace} {\sigma ^{2}_{y} }} }} \right]}\,{\text{or}}\,\alpha = \frac{{n\overline{\rho } }} {{1 + \overline{\rho } {\left( {n - 1} \right)}}}, $$

where $\overline{\rho }$ is equal to the mean inter-item correlation.

Cronbach’s alpha is a generalization of a KR20 Kuder–Richardson formula number 20 coefficient to estimate the reliability of items scored dichotomously with zero or one (Kuder and Richardson 1937). The KR20 is computed as follows:

$$ {\text{KR}}20 = \frac{n} {{n - 1}}{\left[ {1 - {\sum\limits_i {p_{i} {{\left( {1 - p_{i} } \right)}} \mathord{\left/ {\vphantom {{{\left( {1 - p_{i} } \right)}} {\sigma ^{2}_{y} }}} \right. \kern-\nulldelimiterspace} {\sigma ^{2}_{y} }} }} \right]}, $$

where n is the number of dichotomous items; p _i is the proportion responding “positively” (i.e., coded to 1) to item i; and $\sigma ^{2}_{y}$ is equal to the variance of the total composite. The KR20 has the same interpretation as Cronbach’s alpha.

To illustrate the computation of Cronbach’s alpha coefficient, let us use a case with 676 ratings at ML 2. Five KPAs at ML 2 are Requirements Management, Software Project Planning, Software Project Tracking and Oversight, Software Quality Assurance, and Software Configuration Management. The sample variances of the ith KPA, $\sigma ^{2}_{i}$, is computed as ${\sum\limits_i {{{\left( {x_{{ij}} - \overline{x} _{i} } \right)}^{2} } \mathord{\left/ {\vphantom {{{\left( {x_{{ij}} - \overline{x} _{i} } \right)}^{2} } {{\left( {N - 1} \right)},}}} \right. \kern-\nulldelimiterspace} {{\left( {N - 1} \right)},}} }$ where the $x_{{ij}} \prime s$ are the codified actual values of ratings for each assessment j of the ith KPA (i = 1,..., 5; j = 1,...,N), $\overline{x} _{i}$ is the mean of all assessments of the ith KPA, and N is the total number of assessments. As an example of data set 1 in Table 2, the computed sample variance for each KPA is 0.066 (Requirements Management), 0.095 (Software Project Planning), 0.098 (Software Project Tracking and Oversight), 0.106 (Software Quality Assurance), and 0.093 (Software Configuration Management). The sum of the sample variance, ${\sum\limits_{i = 1}^5 {\sigma ^{2}_{i} } }$, is 0.458. For the five-KPAs, the sample variance of the sum of the five KPAs is 1.904 ${\left( {\sigma ^{2}_{y} } \right)}$. Then, Cronbach’s alpha coefficient would be 0.95, i.e., (5 / 4) × (1 − 0.458 / 1.904).

Appendix 2

1.1 Convergent and Discriminant Validities

Figure 7 is a redrawing of Fig. 6b that includes the component (factor) loadings to explain convergent and discriminant validities, where λ _ij is the component loading to an indicator; ɛ _ij and δ _ij denotes measurement error of indicators (KPAs) in exogenous and indigenous latent variables (constructs), respectively. The notations ξ and η are exogenous and indigenous latent variables, respectively. In addition, ζ denotes an error term of indigenous latent variable.

Fig. 7

Full presentation of results for the theoretical model in Fig. 6b

PLS modeling including other SEMs consists of two components. The first is the inner model (also termed inner relationships, structural models, or substantive theory). For example, the relationship between the latent construct “Project Implementation” (ML 2) and “Organization Implementation” (ML 3) in Figure 7 is an inner model. The latter is the outer model (also referred to as measurement model or outer relationships) that defines how each block of indicators relates to its latent variables.

Convergent (also referred to as composite reliability) and discriminant validities are components of a measurement concept known as construct validity. These two validities summarize how well indicators relate to the constructs (Gefen and Straub 2005). They can be evaluated within the PLS model.

Convergent validity is the extent to which multiple methods to construct measurement provide the same results (Campbell and Fiske 1959). Each indicator can be viewed as a different method of measuring the same construct, so the extent of convergence of the set of indicators on a construct can be assessed by showing that each indicator strongly correlates with its assumed theoretical construct. Convergent validity in a given block of indicators is calculated as follows (Werts et al. 1974):

$$ \rho _{c} = \frac{{{\left( {{\sum\limits_i {\lambda _{{ij}} } }} \right)}^{2} }} {{{\left( {{\sum\limits_i {\lambda _{{ij}} } }} \right)}^{2} + {\sum\limits_i {\operatorname{var} {\left( {\varepsilon _{{ij}} } \right)}} }}}\,\,{\text{for}}\,{\text{construct}}\,j, $$

where ɛ _ij is replaced with δ _ij for the block of indicators in the exogenous latent variables. A low value of ρ _c means poor construct definition and/or multidimensionality (where more than one defined construct may be preferable). A recommended value of the coefficient is greater than 0.7 (“modest composite reliability”) (Nunnally and Bernstein 1994).

Discriminant validity of a set of constructs can be undertaken after the convergent validity of individual constructs has been established. Discriminant validity assesses the extent to which a construct and its indicators differ from another construct and its indicators. Discriminant validity is shown when each indicator weakly or not significantly correlates with other constructs in the same model (i.e., the variance in the indicator mostly reflect the variance attributable to its assumed latent variable and not to other latent variables) (Gefen and Straub 2005). Average variance extracted (AVE) is used to summarize discriminant validity (Fornell and Larcker 1981). It is computed as follows:

$$ {\text{AVE}} = \frac{{{\sum\limits_i {\lambda ^{2}_{{ij}} } }}} {{{\sum\limits_i {\lambda ^{2}_{{ij}} } } + {\sum\limits_i {\operatorname{var} {\left( {\varepsilon _{{ij}} } \right)}} }}}\,\,{\text{for}}\,{\text{construct}}\,j. $$

As a rule of thumb, criteria for determining discriminant validity are that the root of the AVE of each construct should be much larger, although there are no guidelines about how much larger, than the correlation of the specific construct with any of the other constructs in the model (Chin 1998). Fornell and Larcker (1981) suggested that AVE can also be interpreted as a measure of reliability for the latent variable component score with a recommended value of 0.5.