Iterative algorithms of biased estimation methods in binary logistic regression

Abstract

Logistic regression is a widely used method to model categorical response data, and maximum likelihood (ML) estimation has widespread use in logistic regression. Although ML method is the most used method to estimate the regression coefficients in logistic regression model, multicollinearity seriously affects the ML estimator. To remedy the undesirable effects of multicollinearity, estimators alternative to ML are proposed. Drawing on the similarities between the multiple linear and logistic regressions, ridge, Liu and two parameter estimators are proposed which are based on the ML estimator. On the other hand, first-order approximated ridge estimator is proposed for use in logistic regression. This study will present further solutions to the problem in the form of alternative estimators which reduce the effect of collinearity. Owing to this, first-order approximated Liu, iterative Liu and iterative two parameter estimators are proposed. A simulation study as well as real life application are carried out to ascertain the effect of sample size and degree of multicollinearity, in which the ML based, first-order approximated and iterative biased estimators are compared. Graphical representations are presented which support the effect of the shrinkage parameter on the mean square error and prediction mean square error of the biased estimators.

Appendices

Appendix 1: Singular value decomposition of $X^{\prime }\hat{W}X$

Using SVD, $X^{\prime }\hat{W}X$ can be expressed as $X^{\prime }\hat{W} X=T\varLambda T^{\prime }$ where $\varLambda =diag(\lambda _{j})$ is a $p\times p$ diagonal matrix of the eigenvalues of $X^{\prime }\hat{W}X\,(\lambda _{1}=\lambda _{\max }\ge \lambda _{2}\ge \cdots \ge \lambda _{p}=\lambda _{\min }$) and $T=[ t_{1},\ldots , t_{p} ]$ is a $p\times p$ orthogonal matrix with columns constitute the eigenvectors (see also Marx and Smith 1990; Aguilera et al. 2006). Then logit is expressed as $X\beta =XTT^{\prime }\beta =Z\alpha$ where $Z=XT,\, \alpha =T^{\prime }\beta .$

Appendix 2: Proof of theorem

The ith element of $\widehat{\alpha }_{kd}^{ML}=T^{\prime }\hat{\beta } _{kd}^{ML}$ can be written as $\widehat{\alpha }_{kd,i}^{ML}=\frac{\lambda _{i}+kd}{\lambda _{i}+k}\widehat{\alpha }_{ML,i}$ where $\widehat{\alpha } _{ML}=T^{\prime }\hat{\beta }_{ML}.$ Then the asymptotic MSE of $\widehat{ \alpha }_{\tilde{k}\hat{d}_{two},i}^{ML}$ equals

$$\begin{aligned} E\left[ \left( \widehat{\alpha }_{\tilde{k}\hat{d}_{two},i}^{ML}-\alpha _{i}\right) ^{2} \right]= & {} E\left[ \left( \frac{\lambda _{i}+\tilde{k}\hat{d}_{two}}{\lambda _{i}+ \tilde{k}}\widehat{\alpha }_{ML,i}-\alpha _{i}\right) ^{2}\right] \nonumber \\= & {} E\left[ \left( \widehat{\alpha }_{ML,i}-\alpha _{i}\right) ^{2}\right] +\frac{2\tilde{k }}{\lambda _{i}+\tilde{k}}E\left[ \left( \widehat{\alpha }_{ML,i}-\alpha _{i}\right) \left( \hat{d}_{two}-1\right) \widehat{\alpha }_{ML,i}\right] \nonumber \\&\quad +\frac{\tilde{k}^{2}}{( \lambda _{i}+\tilde{k}) ^{2}}E\left[ \left( \hat{d}_{two}-1\right) ^{2}\widehat{\alpha }_{ML,i}^{2}\right] . \end{aligned}$$

(10)

To use lemma, let

$$\begin{aligned}&A=E\left[ \left( \widehat{\alpha }_{ML,i}-\alpha _{i}\right) \tilde{k}\left( \hat{d}_{two}-1\right) \widehat{\alpha }_{ML,i}\right] \\&\quad ={-}h_{two}\tilde{k}E\left[ \left( \widehat{\alpha } _{ML,i}-\alpha _{i}\right) \frac{\sum \nolimits _{j=1}^{p}\frac{1}{\lambda _{j}(\lambda _{j}+\tilde{k})}}{\sum \nolimits _{j=1}^{p}\frac{\tilde{k}\widehat{\alpha } _{ML,j}^{2}}{(\lambda _{j}+\tilde{k})^{2}}}\widehat{\alpha }_{ML,i}\right] , \end{aligned}$$

and $f(\widehat{\alpha }_{ML,i})=\frac{\sum \nolimits _{j=1}^{p}\frac{1}{\lambda _{j}(\lambda _{j}+\tilde{k})}}{\sum \nolimits _{j=1}^{p}\frac{\tilde{k}\widehat{ \alpha }_{ML,j}^{2}}{(\lambda _{j}+\tilde{k})^{2}}}\widehat{\alpha }_{ML,i}.$ On account of $\widehat{\alpha }_{ML,i}\sim N(\alpha _{i},\,\lambda _{i}^{-1})$ (see Marx and Smith 1990, p. 28), we get

$$\begin{aligned} A={-}h_{two}\tilde{k}\frac{1}{\lambda _{i}}E\left[ \frac{\sum \limits _{j=1}^{p} \frac{1}{\lambda _{j}(\lambda _{j}+\tilde{k})}}{\sum \nolimits _{j=1}^{p}\frac{ \tilde{k}\widehat{\alpha }_{ML,j}^{2}}{(\lambda _{j}+\tilde{k})^{2}}}-\frac{2 \tilde{k}\widehat{\alpha }_{ML,i}^{2}}{(\lambda _{i}+\tilde{k})^{2}}\frac{ \sum \nolimits _{j=1}^{p}\frac{1}{\lambda _{j}(\lambda _{j}+\tilde{k})}}{\left[ \sum \nolimits _{j=1}^{p}\frac{\tilde{k}\widehat{\alpha }_{ML,j}^{2}}{(\lambda _{j}+\tilde{k})^{2}}\right] ^{2}}\right] . \end{aligned}$$

Using A,  the expression (9) can be presented as

(11)

From Eq. (10), the difference $\varDelta =sMSE(\widehat{\alpha }_{\tilde{k}\hat{ d}_{two}}^{ML})-sMSE(\widehat{\alpha }_{ML})$ equals

$$\begin{aligned} \varDelta= & {} E\left\{ \frac{\sum \nolimits _{j=1}^{p}\frac{1}{\lambda _{j}(\lambda _{j}+ \tilde{k})}}{\sum \nolimits _{j=1}^{p}\frac{\tilde{k}\widehat{\alpha }_{ML,j}^{2} }{(\lambda _{j}+\tilde{k})^{2}}}\left[ \sum \limits _{i=1}^{p}\frac{-2h_{two}\tilde{k }}{\lambda _{i}(\lambda _{i}+\tilde{k})}+\sum \limits _{i=1}^{p}\frac{4h_{two} \tilde{k}^{2}\widehat{\alpha }_{ML,i}^{2}}{\lambda _{i}(\lambda _{i}+\tilde{k })^{3}}\frac{1}{\sum \nolimits _{j=1}^{p}\frac{\tilde{k}\widehat{\alpha } _{ML,j}^{2}}{(\lambda _{j}+\tilde{k})^{2}}}\right. \right. \\&\left. \left. \quad +\, h_{two}^{2}\tilde{k}^{2}\sum \limits _{i=1}^{p}\frac{\widehat{\alpha } _{ML,i}^{2}}{(\lambda _{i}+\tilde{k})^{2}}\frac{\sum \nolimits _{j=1}^{p}\frac{1 }{\lambda _{j}(\lambda _{j}+\tilde{k})}}{\sum \nolimits _{j=1}^{p}\frac{\tilde{k} \widehat{\alpha }_{ML,j}^{2}}{(\lambda _{j}+\tilde{k})^{2}}}\right] \right\} . \end{aligned}$$

Therefore,

$$\begin{aligned} \varDelta\le & {} E\left\{ \frac{\sum \nolimits _{j=1}^{p}\frac{1}{\lambda _{j}(\lambda _{j}+\tilde{k})}}{\sum \nolimits _{j=1}^{p}\frac{\tilde{k}\widehat{\alpha } _{ML,j}^{2}}{(\lambda _{j}+\tilde{k})^{2}}}h_{two}\tilde{k} \left[ \sum \limits _{j=1}^{p}\frac{-2}{\lambda _{j}(\lambda _{j}+\tilde{k})}+\frac{4 }{\lambda _{\min }(\lambda _{\min }+\tilde{k})}+h_{two}\sum \limits _{j=1}^{p} \frac{1}{\lambda _{j}(\lambda _{j}+\tilde{k})}\right] \right\} \\= & {} E\left[ \frac{\sum \nolimits _{j=1}^{p}\frac{1}{\lambda _{j}(\lambda _{j}+ \tilde{k})}}{\sum \nolimits _{j=1}^{p}\frac{\tilde{k}\widehat{\alpha }_{ML,j}^{2} }{(\lambda _{j}+\tilde{k})^{2}}}\right] h_{two}\tilde{k}\left[ \sum \limits _{j=1}^{p}\frac{-2}{\lambda _{j}(\lambda _{j}+\tilde{k})}+\frac{4}{ \lambda _{\min }(\lambda _{\min }+\tilde{k})}+h_{two}\sum \limits _{j=1}^{p} \frac{1}{\lambda _{j}(\lambda _{j}+\tilde{k})}\right] . \end{aligned}$$

For $\varDelta$ to be negative, $H(h_{two})=h_{two}\left[ \sum \nolimits _{j=1}^{p} \frac{-2}{\lambda _{j}(\lambda _{j}+\tilde{k})}+\frac{4}{\lambda _{\min }(\lambda _{\min }+\tilde{k})}\right.$ $\left. +h_{two}\sum \nolimits _{j=1}^{p}\frac{1}{\lambda _{j}(\lambda _{j}+\tilde{k})}\right]$ should be negative. $H(h_{two})$ is negative for $0<h_{two}<2\left[ 1-\frac{2}{\sum \nolimits _{j=1}^{p}\frac{ \lambda _{\min }(\lambda _{\min }+\tilde{k})}{\lambda _{j}(\lambda _{j}+ \tilde{k})}}\right]$ if $\sum \nolimits _{j=1}^{p}\frac{\lambda _{\min }(\lambda _{\min }+\tilde{k})}{\lambda _{j}(\lambda _{j}+\tilde{k})}>2$ and for $2\left[ 1-\frac{2}{\sum \nolimits _{j=1}^{p}\frac{\lambda _{\min }(\lambda _{\min }+\tilde{k})}{\lambda _{j}(\lambda _{j}+\tilde{k})}}\right]<h_{two}<0$ if $\sum \nolimits _{j=1}^{p}\frac{\lambda _{\min }(\lambda _{\min }+\tilde{k})}{\lambda _{j}(\lambda _{j}+\tilde{k})}<2.$ Since we look for $\hat{d}_{two}$ in interval (0, 1), the proof is completed.