Residual plots to reveal the functional form for covariates in parametric accelerated failure time models

Abstract

We study residual plots for parametric accelerated failure time (AFT) models, using both standardized residuals and Cox-Snell residuals. Two different approaches are discussed in the case of censored data; adjusting censored residuals by adding a residual time, and using nonparametric exponential regression of non-adjusted censored Cox-Snell residuals. The main object of the paper is to show how residuals can be used to infer the correct functional form for possibly misspecified covariates. We demonstrate the use of the methods by analysis of two reliability data sets and by a simulation study using Weibull-distributed data. We also consider briefly a corresponding approach for parametric proportional hazards models.

Appendix: More on parameters of misspecified models

Following White (1982), the starred parameters defined in Sect. 5 are found by minimizing the expected value of the log of the ratio between the density for $(T,\varDelta ,{\varvec{X}})$ under the true model and under the misspecified model, when the random variables themselves, having the true distribution, are used in the densities. It is here natural to assume that the distribution of ${\varvec{X}}$ and the conditional distribution of $C$ given ${\varvec{X}}$ are the same for both models, so we compare in effect the densities

$$\begin{aligned} g(t|{\varvec{x}})^\delta G(t|{\varvec{x}})^{1-\delta } \end{aligned}$$

(34)

corresponding to the two models.

Under the assumptions for AFT models, the general expressions for the functions in (34) are

$$\begin{aligned} g(t|{\varvec{x}})&= \frac{1}{t\sigma } \phi \left( \frac{\log t - f({\varvec{x}})}{\sigma } \right) \\ G(t|{\varvec{x}})&= 1- \varPhi \left( \frac{\log t - f({\varvec{x}})}{\sigma } \right) \end{aligned}$$

From this, and substituting the assumed form of $f({\varvec{x}})$ for the two models, we can write the criterion to be minimized as

$$\begin{aligned} E(D)&\equiv E \left[ \varDelta \;\log \frac{ \frac{1}{\sigma }\phi \left( \frac{\log T - \beta _0 - {\varvec{\beta }}'Z - f(X)}{\sigma } \right) }{ \frac{1}{\sigma ^*}\phi \left( \frac{\log T - \beta ^*_0 - {{\varvec{\beta }}^*}'Z - \gamma ^* X}{\sigma ^*} \right) }\right. \nonumber \\&\quad \left. + (1-\varDelta ) \; \log \frac{1-\varPhi \left( \frac{\log T - \beta _0 - {\varvec{\beta }}'Z - f(X)}{\sigma } \right) }{1-\varPhi \left( \frac{\log T - \beta ^*_0 - {{\varvec{\beta }}^*}'Z - \gamma ^* X}{\sigma ^*} \right) } \right] \nonumber \\&= E \left[ \varDelta \log \left\{ \frac{\sigma ^*}{\sigma } \frac{ \phi (W)}{ \phi \left( \frac{\sigma }{\sigma ^*}W +\frac{(\beta _0 -\beta ^*_0) + ({{\varvec{\beta }} - {\varvec{\beta ^*}}})'{{\varvec{z}}} + f(x) - \gamma ^* x }{\sigma ^*} \right) } \right\} \right] \nonumber \\&\quad + E \left[ (1- \varDelta ) \log \frac{ 1- \varPhi (W)}{ 1-\varPhi \left( \frac{\sigma }{\sigma ^*}W +\frac{(\beta _0 -\beta ^*_0) + ({{\varvec{\beta }} - {\varvec{\beta ^*}}})'{{\varvec{z}}} + f(x) - \gamma ^* x }{\sigma ^*} \right) } \right] \qquad \end{aligned}$$

(35)

where expectation is taken with respect to the true joint distribution of $({\varvec{X}},W,C)$, where ${\varvec{X}}=(X,{\varvec{Z}})$. Note also that we can express $\varDelta$ in terms of $({\varvec{X}},W,C)$ as $\varDelta = I( \beta _0 + {\varvec{\beta }}'{\varvec{Z}}+ f(X) + \sigma W < C)$. The task is to minimize the expression in (35) with respect to the starred parameters. From the expression it can be seen that the minimizing parameters may depend on the censoring distribution, which is an interesting observation since it is well known that for a correctly specified distribution, the maximum likelihood estimators under independent censoring will always converge to the true model independently of the censoring scheme.

In general the minimization of (35) may be difficult to do analytically. A simple way of “cheating” to get approximate values for the starred parameters is to simulate from the true model a (very) large number of observations and then use a statistical package (e.g. R) to compute the maximum likelihood estimators (see Sect. 7). Here we shall for illustration go through some examples of how (35) will look in particular cases, and in some cases we also show how it can be minimized analytically.

1.1 Example—lognormal distribution

Assume here that $W$ has the standard normal distribution, and that there is no censoring. Assume for simplicity that $Z$ is one-dimensional and assume without loss of generality that $E(X)=E(Z)=0$.

From (35) with $P(\varDelta =1)=1$, with $\phi$ being the standard normal density and $W$ being standard normally distributed, we have

$$\begin{aligned} E(D) = \log \frac{\sigma ^*}{\sigma } -\frac{1}{2} + \frac{\sigma ^2}{2\sigma ^{*2}} +\frac{E\left\{ (\beta _0- \beta _0^* + (\beta _1 - \beta _1^*)Z + f(X)- \gamma ^*X)^2\right\} }{2 \sigma ^{*2}}. \end{aligned}$$

By differentiation with respect to all the starred parameters we obtain the solutions

$$\begin{aligned} \beta _0^* - \beta _0&= Ef(X)\\ \beta _1^* - \beta _1&= \frac{E\{ Z f(X) \} - E\{XZ\} E\{Xf(X)\}}{1-(E\{XZ\})^2} \\ \gamma ^*&= \frac{E\{ X f(X) \} - E\{XZ\} E\{Zf(X)\}}{1-(E\{XZ\})^2}\\ \sigma ^*&= \sqrt{\sigma ^2 + M} , \end{aligned}$$

where $M$ is the minimized value of $E\{(\beta _0- \beta _0^* + (\beta _1 - \beta _1^*)Z + f(X)- \gamma ^*X)^2\}$.

Suppose now that $X,Z$ are independent, and assume (without loss of generality) that also $Ef(X)=0$. Then the solution is

$$\begin{aligned} \beta _0^* - \beta _0&= 0\\ \beta _1^* - \beta _1&= 0 \\ \gamma ^*&= E\{ X f(X) \} = Cov(X,f(X))\\ \sigma ^*&= \sqrt{\sigma ^2 + E\{ (f(X)-\gamma ^*X)^2 \}} . \end{aligned}$$

Suppose now instead that $Cov(X,Z) \equiv E(XZ) = \rho$, but still $E(X)=E(Z)=E(f(X))=0$, while also assuming (without loss of generality) that $E(X^2)=E(Z^2)=1$. Then we get

$$\begin{aligned} \beta _0^* - \beta _0&= 0\\ \beta _1^* - \beta _1&= \frac{E\{ (Z -\rho X)f(X) \} }{1-\rho ^2} \\ \gamma ^*&= \frac{E\{ (X -\rho Z)f(X) \} }{1-\rho ^2}. \end{aligned}$$

Note that $Cov(Z-\rho X, X)=0$, so the numerator of the expression for $\beta _1^* - \beta _1$ is the expected value of a product of something that is uncorrelated with $X$ times a function of $X$. Intuitively this should be a small number. In fact, if we further assume that $(X,Z)$ is binormal, and that $f(X)=X^2-E(X^2)$, then since $Var(X|Z)=1-\rho ^2$ and $E(X|Z)=\rho Z$, we get

$$\begin{aligned} E(ZX^2)&= E[ZE(X^2|Z)] = E[Z (Var(X|Z)+(E(X|Z))^2)]\\&= E[Z(1-\rho ^2+\rho ^2Z^2)] = 0, \end{aligned}$$

which in fact implies that $\beta _1^* - \beta _1 =0$ in this case.

1.2 Example—Weibull distribution

Now $\phi (x)=e^x e^{-e^x}$ and $\varPhi (x)=1- e^{-e^x}$, so $\log \phi (x) = x-e^x$ and $\log (1-\varPhi (x) = -e^x$. Note also that $Ee^W=1$. If we allow censoring, we get

$$\begin{aligned} E(D) \!&= \! E \left[ \varDelta \log \frac{\sigma ^*}{\sigma } \!+\! \varDelta W \!-\! 1 \!-\! \varDelta \left( \frac{\sigma }{\sigma ^*} W \!+\! \frac{\beta _0- \beta _0^* \!+\! ({\varvec{\beta }}_1 \!-\! {\varvec{\beta }}_1^*)'{\varvec{Z}}\!+\! f(X)\!-\! \gamma ^*X}{\sigma ^*}\right) \right. \\&\quad + \left. \exp { \left\{ \frac{\sigma }{\sigma ^*} W + \frac{\beta _0- \beta _0^* + ({\varvec{\beta }}_1 - {\varvec{\beta }}_1^*)'{\varvec{Z}}+ f(X)- \gamma ^*X}{\sigma ^*}\right\} }\right] . \end{aligned}$$

The last term equals (27) and is hence conditionally Weibull distributed given ${\varvec{Z}}$ and $X$, with conditional expectation

$$\begin{aligned} \varGamma \left( 1+\frac{\sigma }{\sigma ^*}\right) \exp \left\{ \frac{\beta _0- \beta _0^* + ({\varvec{\beta }}_1 - {\varvec{\beta }}_1^*)'{\varvec{Z}}+ f(X)- \gamma ^*X}{\sigma ^*} \right\} . \end{aligned}$$

The above formula for $E(D)$ furthermore involves $E(\varDelta )=P(\beta _0+{\varvec{\beta _1}}'{\varvec{Z}}+f(X)+\sigma W < C)$ and also $E(\varDelta W)$, which may be fairly complicated expressions.

Let us therefore below consider the non-censored case. Note here that $E(W)=-a$, where $a= 0.577215665\ldots$ is Euler’s constant. The expression then becomes

$$\begin{aligned} E(D) \!&= \! \log \frac{\sigma ^*}{\sigma } \!-\! a \left( 1\!-\!\frac{\sigma }{\sigma ^*}\right) \!-\! 1 \!-\! \frac{\beta _0- \beta _0^* \!+\! ({\varvec{\beta }}_1 \!-\! {\varvec{\beta _1}}^*)'E({\varvec{Z}}) \!+\! E(f(X)) \!-\! \gamma ^*E(X)}{\sigma ^*} \\&\quad + \varGamma \left( 1\!+\!\frac{\sigma }{\sigma ^*}\right) E \left[ \exp \left\{ \frac{\beta _0- \beta _0^* \!+\! ({\varvec{\beta }}_1 \!-\! {\varvec{\beta }}_1^*)'{\varvec{Z}}\!+\! f(X) - \gamma ^*X}{\sigma ^*} \right\} . \right] \end{aligned}$$

Now if we assume that $X,{\varvec{Z}}$ are independent, with ${\varvec{Z}}$ being multinormally distributed with covariance matrix given by the identity matrix, while $E(X)=E(f(X))=0$ (but $X$ not necessarily normal), then $E(D)$ becomes

$$\begin{aligned} \log \frac{\sigma ^*}{\sigma }&- a \left( 1-\frac{\sigma }{\sigma ^*}\right) - 1 - \frac{\beta _0- \beta _0^*}{\sigma ^*}\\&+ \varGamma \left( 1+\frac{\sigma }{\sigma ^*}\right) \exp \left\{ \frac{\beta _0- \beta _0^*}{\sigma ^*} + \frac{({\varvec{\beta }}_1- {\varvec{\beta }}_1^*)'({\varvec{\beta }}_1- {\varvec{\beta }}_1^*) }{2\sigma ^{*2}}\right\} \\&\times E \exp \left\{ \frac{f(X)-\gamma ^* X}{\sigma ^*} \right\} . \end{aligned}$$

It is clear from this expression that the solution for ${\varvec{\beta }}_1^*$ is ${\varvec{\beta }}_1^*-{\varvec{\beta }}_1=0$, but note that multinormality of ${\varvec{Z}}$ is crucial for this result. No explicit solution can be found for the other parameters, however, so numerical methods are needed. But it can be seen that if (i) $X$ has a distribution that is symmetric around 0, i.e. $X$ and $-X$ has the same distribution, and (ii) $f(X)= g(X) - E(g(X))$ where $g(-x)=g(x)$, then $\gamma ^*=0$. To see this, differentiate the above expression with respect to $\gamma ^*$ and check that the result equals 0 if $\gamma ^*$ is set to 0.