Review of Issues About Classical Change Scores: A Multilevel Modeling Perspective on Some Enduring Beliefs

Abstract

Change scores obtained in pretest–posttest designs are important for evaluating treatment effectiveness and for assessing change of individual test scores in psychological research. However, over the years the use of change scores has raised much controversy. In this article, from a multilevel perspective, we provide a structured treatise on several persistent negative beliefs about change scores and show that these beliefs originated from the confounding of the effects of within-person change on change-score reliability and between-person change differences. We argue that psychometric properties of change scores, such as reliability and measurement precision, should be treated at suitable levels within a multilevel framework. We show that, if examined at the suitable levels with such a framework, the negative beliefs about change scores can be renounced convincingly. Finally, we summarize the conclusions about change scores to dispel the myths and to promote the potential and practical usefulness of change scores.

Appendices

Appendix A

Equation (11) models the true pretest score $\tau _{v1}$ and the true posttest score $\tau _{v1}+\delta _v$. Alternatively, we may model the true pretest score $\tau _{v1}$ and the true change $\delta _v$ (rather than the true posttest score) as follows:

$$\begin{aligned} \begin{bmatrix} \tau _{v1}\\ \delta _v \end{bmatrix}=\begin{bmatrix} \mu _{\tau _1}\\ \mu _\delta \end{bmatrix}+\begin{bmatrix} \upsilon _1\\ \upsilon _2 \end{bmatrix}, \end{aligned}$$

(A1)

with

$$\begin{aligned} \begin{bmatrix} \upsilon _1\\ \upsilon _2 \end{bmatrix} \sim N(\mathbf {0}, \mathbf {\Sigma }_\upsilon ), \end{aligned}$$

(A2)

where $\mathbf {\Sigma }_\upsilon =\begin{bmatrix} \sigma ^2_{\tau _1}&\sigma _{\tau _1\delta }\\ \sigma _{\tau _1\delta }&\sigma ^2_\delta \end{bmatrix}$. $\sigma ^2_{\tau _1}$ denotes the variance of the true pretest score. $\sigma _{\tau _1\delta }$ denotes the covariance between the true pretest score and true change. $\sigma ^2_\delta$ denotes the variance of true change. Given (A1) and (A2), we can derive the variance of the true posttest score, denoted by $\sigma ^2_{\tau _{2}}$, and the covariance between the true pretest score and the true posttest score, denoted by $\sigma _{\tau _{1}\tau _{2}}$, as follows:

$$\begin{aligned} \sigma ^2_{\tau _{2}}=\sigma ^2_{\tau _1}+2\sigma _{\tau _1\delta }+\sigma ^2_\delta , \end{aligned}$$

(A3)

and

$$\begin{aligned} \sigma _{\tau _{1}\tau _{2}}=\sigma ^2_{\tau _1}+\sigma _{\tau _1\delta }. \end{aligned}$$

(A4)

Note that in Eq. (13) the covariance matrix $\mathbf {\Sigma }_\omega$ can be expressed as

$$\begin{aligned} \mathbf {\Sigma }_\omega =\begin{bmatrix} \sigma ^2_{\tau _{1}}&\sigma _{\tau _{1}\tau _{2}}\\ \sigma _{\tau _{1}\tau _{2}}&\sigma ^2_{\tau _{2}} \end{bmatrix}, \end{aligned}$$

(A5)

and given (A3) and (A4), we thus can derive

$$\begin{aligned} \mathbf {\Sigma }_\omega = \begin{bmatrix} \sigma ^2_{\tau _1}&\sigma ^2_{\tau _1}+\sigma _{\tau _1\delta }\\ \sigma ^2_{\tau _1}+\sigma _{\tau _1\delta }&\sigma ^2_{\tau _1}+2\sigma _{\tau _1\delta }+\sigma ^2_\delta \end{bmatrix}. \end{aligned}$$

(A6)

Appendix B

Let $\sigma ^2_1$ denote the variance of the observed pretest score, $\sigma ^2_2$ denote the variance of the observed posttest score, and let $\sigma _{12}$ denote the covariance between the pretest score and the posttest score. Then, the variance of change scores $\sigma ^2_D$ is

$$\begin{aligned} \sigma ^2_D=\sigma ^2_2-2\sigma _{12}+\sigma ^2_1. \end{aligned}$$

(B1)

According to Eq. (16),

$$\begin{aligned} \sigma ^2_2= & {} \sigma ^2_\delta +2\sigma _{\tau _1\delta }+\sigma ^2_{\tau _1}+\sigma ^2_{\varepsilon _2}, \end{aligned}$$

(B2)

$$\begin{aligned} \sigma _{12}= & {} \sigma _{\tau _1\delta }+\sigma ^2_{\tau _1}+\sigma _{\varepsilon _1\varepsilon _2}, \end{aligned}$$

(B3)

and

$$\begin{aligned} \sigma ^2_1=\sigma ^2_{\varepsilon _1}+\sigma ^2_{\tau _1}. \end{aligned}$$

(B4)

Thus, replacing the right-hand side of Eq. (B1) with (B2), (B3), and (B4), we obtain

$$\begin{aligned} \sigma ^2_D = \sigma ^2_\delta +\sigma ^2_{\varepsilon _1} + \sigma ^2_{\varepsilon _2} -2\sigma _{\varepsilon _1\varepsilon _2} \end{aligned}$$

(B5)

as desired.

Appendix C

Here, it suffices to show that it is possible to obtain positive values for $\rho _{DD'}-\rho _{11'}$ and $\rho _{DD'}-\rho _{22'}$ theoretically. Whether $\rho _{DD'}-\rho _{11'}>0$ and $\rho _{DD'}-\rho _{22'}>0$ are observed in empirical studies is irrelevant.

$\rho _{DD'}-\rho _{11'}$ can be derived as follows. Given Eqs. (17) and (19),

$$\begin{aligned} \begin{aligned} \rho _{DD'}-\rho _{11'}&= \frac{\sigma ^2_\delta \sigma ^2_{\varepsilon _1}-\sigma ^2_{\tau _1}(\sigma ^2_{\varepsilon _1}+ \sigma ^2_{\varepsilon _2}-2\sigma _{\varepsilon _1\varepsilon _2})}{(\sigma ^2_\delta +\sigma ^2_{\varepsilon _1}+\sigma ^2_{\varepsilon _2}-2\sigma _{\varepsilon _1\varepsilon _2})(\sigma ^2_{\tau _1}+\sigma ^2_{\varepsilon _1})} \end{aligned} \end{aligned}$$

(C1)

According to the Cauchy–Schwarz inequality,

$$\begin{aligned} \sigma _{\varepsilon _1\varepsilon _2}\le \sigma _{\varepsilon _1}\sigma _{\varepsilon _2}, \end{aligned}$$

which implies that

$$\begin{aligned} \sigma ^2_{\varepsilon _1}+\sigma ^2_{\varepsilon _2}-2\sigma _{\varepsilon _1\varepsilon _2}\ge \sigma ^2_{\varepsilon _1}+\sigma ^2_{\varepsilon _2} -2\sigma _{\varepsilon _1}\sigma _{\varepsilon _2}=(\sigma _{\varepsilon _1}-\sigma _{\varepsilon _2})^2. \end{aligned}$$

This means that the denominator of (C1) is always positive. The denominator can equal 0, when $\sigma _{\varepsilon _1}=\sigma _{\varepsilon _2}=\sigma _{\tau _1}=\sigma _{\delta }=0$. The numerator of (C1) can be positive as well, as long as (for example) $\sigma ^2_\delta$ is high enough (keeping everything else constant) so that $\sigma ^2_\delta \sigma ^2_{\varepsilon _1}>\sigma ^2_{\tau _1}(\sigma ^2_{\varepsilon _1}+ \sigma ^2_{\varepsilon _2}-2\sigma _{\varepsilon _1\varepsilon _2})$. Thus, we have shown that $\rho _{DD'}-\rho _{11'}>0$ is possible.

$\rho _{DD'}-\rho _{22'}$ can be derived as follows. Given Eqs. (17) and (20),

$$\begin{aligned} \begin{aligned} \rho _{DD'}-\rho _{22'}&=\frac{\sigma ^2_\delta }{\sigma ^2_\delta +\sigma ^2_{\varepsilon _1}+\sigma ^2_{\varepsilon _2}-2\sigma _{\varepsilon _1\varepsilon _2}} - \frac{\sigma ^2_{\tau _1}+\sigma ^2_\delta +2\sigma _{\tau _1\delta }}{\sigma ^2_{\tau _1}+\sigma ^2_\delta +2\sigma _{\tau _1\delta }+\sigma ^2_{\varepsilon _2}}\\&= \frac{\sigma ^2_\delta (\sigma ^2_{\tau _1}+\sigma ^2_\delta +2\sigma _{\tau _1\delta }+\sigma ^2_{\varepsilon _2})-(\sigma ^2_{\tau _1}+\sigma ^2_\delta +2\sigma _{\tau _1\delta })(\sigma ^2_\delta +\sigma ^2_{\varepsilon _1}+\sigma ^2_{\varepsilon _2}-2\sigma _{\varepsilon _1\varepsilon _2})}{(\sigma ^2_\delta +\sigma ^2_{\varepsilon _1}+\sigma ^2_{\varepsilon _2}-2\sigma _{\varepsilon _1\varepsilon _2})(\sigma ^2_{\tau _1}+\sigma ^2_\delta +2\sigma _{\tau _1\delta }+\sigma ^2_{\varepsilon _2})}\\&= \frac{(-\sigma ^2_{\varepsilon _1}-\sigma ^2_{\varepsilon _2}+2\sigma _{\varepsilon _1\varepsilon _2})(\sigma ^2_{\tau _1}+2\sigma _{\tau _1\delta })+\sigma ^2_\delta (2\sigma _{\varepsilon _1\varepsilon _2}-\sigma ^2_{\varepsilon _1})}{(\sigma ^2_\delta +\sigma ^2_{\varepsilon _1}+\sigma ^2_{\varepsilon _2}-2\sigma _{\varepsilon _1\varepsilon _2})(\sigma ^2_{\tau _1}+\sigma ^2_\delta +2\sigma _{\tau _1\delta }+\sigma ^2_{\varepsilon _2})}. \end{aligned} \end{aligned}$$

(C2)

We first examine the denominator of (C2). According to the Cauchy–Schwarz inequality,

$$\begin{aligned} -\sigma _{\varepsilon _1}\sigma _{\varepsilon _2}\le \sigma _{\varepsilon _1\varepsilon _2}\le \sigma _{\varepsilon _1}\sigma _{\varepsilon _2}, \end{aligned}$$

and

$$\begin{aligned} -\sigma _{\tau _1}\sigma _{\delta }\le \sigma _{\tau _1\delta }\le \sigma _{\tau _1}\sigma _{\delta }, \end{aligned}$$

and thus, it can be proven that

$$\begin{aligned} \sigma ^2_\delta +\sigma ^2_{\varepsilon _1}+\sigma ^2_{\varepsilon _2}-2\sigma _{\varepsilon _1\varepsilon _2}\ge \sigma ^2_\delta +\sigma ^2_{\varepsilon _1}+\sigma ^2_{\varepsilon _2}-2\sigma _{\varepsilon _1}\sigma _{\varepsilon _2}=\sigma ^2_\delta +(\sigma _{\varepsilon _1}-\sigma _{\varepsilon _2})^2\ge 0, \end{aligned}$$

and

$$\begin{aligned} \sigma ^2_{\tau _1}+\sigma ^2_\delta +2\sigma _{\tau _1\delta }+\sigma ^2_{\varepsilon _2}\ge \sigma ^2_{\tau _1}+\sigma ^2_\delta -2\sigma _{\tau _1}\sigma _\delta +\sigma ^2_{\varepsilon _2}= (\sigma _{\tau _1}-\sigma _\delta )^2+\sigma ^2_{\varepsilon _2}\ge 0. \end{aligned}$$

Therefore, the denominator of (C2) is always positive. The denominator can equal 0, when $\sigma _{\tau _1}=\sigma _{\delta }=\sigma _{\varepsilon _1}=\sigma _{\varepsilon _2}=0$.

We now examine whether the numerator of (C2)—$(-\sigma ^2_{\varepsilon _1}-\sigma ^2_{\varepsilon _2}+2\sigma _{\varepsilon _1\varepsilon _2})(\sigma ^2_{\tau _1}+2\sigma _{\tau _1\delta })+\sigma ^2_\delta (2\sigma _{\varepsilon _1\varepsilon _2}-\sigma ^2_{\varepsilon _1})$—can be positive. According to the Cauchy–Schwarz inequality, $-\sigma ^2_{\varepsilon _1}-\sigma ^2_{\varepsilon _2}+2\sigma _{\varepsilon _1\varepsilon _2}$ cannot be positive:

$$\begin{aligned} -\sigma ^2_{\varepsilon _1}-\sigma ^2_{\varepsilon _2}+2\sigma _{\varepsilon _1\varepsilon _2}\le -\sigma ^2_{\varepsilon _1}-\sigma ^2_{\varepsilon _2} + 2\sigma _{\varepsilon _1}\sigma _{\varepsilon _2} = -(\sigma _{\varepsilon _1}-\sigma _{\varepsilon _2})^2\le 0. \end{aligned}$$

(C3)

Therefore, the numerator of (C2) can be positive due to the following three sufficient conditions: 1) $\sigma ^2_{\tau _1}+2\sigma _{\tau _1\delta }<0$ and $\sigma ^2_\delta (2\sigma _{\varepsilon _1\varepsilon _2}-\sigma ^2_{\varepsilon _1})>0$; 2) $\sigma ^2_{\tau _1}+2\sigma _{\tau _1\delta }<0$, $\sigma ^2_\delta (2\sigma _{\varepsilon _1\varepsilon _2}-\sigma ^2_{\varepsilon _1})<0$, but $(-\sigma ^2_{\varepsilon _1}-\sigma ^2_{\varepsilon _2}+2\sigma _{\varepsilon _1\varepsilon _2})(\sigma ^2_{\tau _1}+2\sigma _{\tau _1\delta })+\sigma ^2_\delta (2\sigma _{\varepsilon _1\varepsilon _2}-\sigma ^2_{\varepsilon _1})>0$; and 3) $\sigma ^2_\delta (2\sigma _{\varepsilon _1\varepsilon _2}-\sigma ^2_{\varepsilon _1})>0$, $\sigma ^2_{\tau _1}+2\sigma _{\tau _1\delta }>0$, but $(-\sigma ^2_{\varepsilon _1}-\sigma ^2_{\varepsilon _2}+2\sigma _{\varepsilon _1\varepsilon _2})(\sigma ^2_{\tau _1}+2\sigma _{\tau _1\delta })+\sigma ^2_\delta (2\sigma _{\varepsilon _1\varepsilon _2}-\sigma ^2_{\varepsilon _1})>0$. The three conditions can happen given suitable values for the parameters. Take the first condition for example, the numerator of (C2) is positive if $\sigma _{\tau _1\delta }<(-1/2)\sigma ^2_{\tau _1}$ and $\sigma _{\varepsilon _1\varepsilon _2}>(1/2)\sigma ^2_{\varepsilon _1}$. Thus, we have shown that $\rho _{DD'}-\rho _{22'}>0$ is possible, given suitable values for $\sigma ^2_{\tau _1}$, $\sigma ^2_{\delta }$, $\sigma ^2_{\varepsilon _1}$, $\sigma ^2_{\varepsilon _2}$, $\sigma _{\tau _1\delta }$, and $\sigma _{\varepsilon _1\varepsilon _2}$.

Appendix D

The covariance between D and $X_1$, denoted by $\sigma _{D1}$, is

$$\begin{aligned} \sigma _{D1} = \sigma _{12} - \sigma ^2_1. \end{aligned}$$

(D1)

Thus, replacing the right-hand side of Eq. (D1) with (B3) and (B4), we obtain

$$\begin{aligned} \sigma _{D1}= \sigma _{\tau _1\delta }+\sigma _{\varepsilon _1\varepsilon _2}-\sigma ^2_{\varepsilon _1}, \end{aligned}$$

(D2)

and hence (D2) is the numerator in Eq. (22).

Appendix E

We assume the variance of true pretest scores is equal to one (i.e., $\sigma ^2_{\tau _1}=1$), which is to identify the scale, and assume 75%, 50%, and 25% of the individuals at the population level have a true change score larger than .5 SD, .75 SD, and 1 SD of the true pretest scores, respectively. Therefore, we need to find a normal distribution with true change scores equal .5, .75, and 1 corresponding to the 25th, 50th and 75th percentiles. Figure 2 presents such a normal distribution of true change scores with $\mu =.75$ and $\sigma ^2=.14$. The variance of this distribution is obtained as follows. For a standard normal distribution, we know that $P(Z>z=.67)=.25$, and therefore, the variance of the true change scores should be $((1-.75)/.67)^2\approx .14$. To see this, let $\sigma$ be the standard deviation of the normal distribution of true change scores. When true score equals 1, if we standardize the true score, then we obtain $(1-.75)/\sigma = .67$. Solving $\sigma$ and taking the square of $\sigma$, we obtain .14.

Fig. 2

A normal distribution of true change scores with $\mu =.75$ and $\sigma ^2=.14$.