Random effects selection in generalized linear mixed models via shrinkage penalty function

Abstract

In this paper, we discuss the selection of random effects within the framework of generalized linear mixed models (GLMMs). Based on a reparametrization of the covariance matrix of random effects in terms of modified Cholesky decomposition, we propose to add a shrinkage penalty term to the penalized quasi-likelihood (PQL) function of the variance components for selecting effective random effects. The shrinkage penalty term is taken as a function of the variance of random effects, initiated by the fact that if the variance is zero then the corresponding variable is no longer random (with probability one). The proposed method takes the advantage of a convenient computation for the PQL estimation and appealing properties for certain shrinkage penalty functions such as LASSO and SCAD. We propose to use a backfitting algorithm to estimate the fixed effects and variance components in GLMMs, which also selects effective random effects simultaneously. Simulation studies show that the proposed approach performs quite well in selecting effective random effects in GLMMs. Real data analysis is made using the proposed approach, too.

Appendix: Derivation of the first- and second-order derivatives

Part 1.

First we show how to derive $\varSigma_{\psi}(\theta_{1}^{(k-1)})$ and $U_{\psi}(\theta_{1}^{(k-1)})$. The main idea is to localize p _ψ (|θ ₁|) approximately by a quadratic function. This is a direct result of Fan and Li (2001). Suppose that we already have $\theta_{1}^{(k-1)}=(\lambda_{1}^{(k-1)},\dots, \lambda_{q}^{(k-1)})$ that is close to the minimizer of (2.5). Then p _ψ (|λ _l |),l=1,…,q can be locally approximated by a quadratic function as

$$\bigl[p_{\psi} \bigl(|\lambda_{l}|\bigr)\bigr]'= p'_{\psi} \bigl(|\lambda_{l}|\bigr)\operatorname{sgn}(\lambda_{l}) \approx \frac{p'_{\psi}(|\lambda_{l}^{(k-1)}|)}{|\lambda_{l}^{(k-1)}|}\lambda_{l} $$

for $\lambda_{l}\approx \lambda_{l}^{(k-1)}$, where $\operatorname{sgn}(\lambda_{l})$ is the sign of λ _l . In other words,

Thus, with the local approximated form of p _ψ (|θ ₁|)=(p _ψ (|λ ₁|),…,p _ψ (|λ _q |)), we can derive its second and first derivatives, i.e., $\varSigma_{\psi}(\theta_{1}^{(k-1)})$ and $U_{\psi}(\theta_{1}^{(k-1)})$ at given value $\theta_{1}^{(k-1)}$.

Part 2.

Without lose of generality, we suppress (k−1) in $\theta_{i}^{(k-1)}$. We will show how to get the elements in ∇l ₂(θ _i ) and ∇² l ₂(θ _i ), and similarly in ∇l _2R (θ _i ) and ∇² l _2R (θ _i ), i=1,2, given θ ₁=(λ ₁,…,λ _q ) and θ ₂=(γ ₂₁;γ ₃₁,γ ₃₂;…;γ _q1,…,γ _q(q−1)).

With $l_{2}(\hat{\beta},\theta)=-\frac{1}{2}\sum_{i=1}^{n}log|V_{i}|-\frac{1}{2}\sum_{i=1}^{n}(Y_{i}-X_{i}\hat{\beta})^{T}$ $V_{i}^{-1}(Y_{i}-X_{i}\hat{\beta})$ and let $e_{i}=Y_{i}-X_{i}\hat{\beta}$, we have

$$ l_2(\hat{\beta},\theta)=-\frac{1}{2}\sum_{i=1}^n\log |V_i|-\frac{1}{2}\sum_{i=1}^ne_i^TV_i^{-1}e_i. $$

(A.1)

Similarly, we have

(A.2)

At first we deal with the elements in ∇l ₂(θ ₁) and ∇² l ₂(θ ₁), i.e., $\frac{\partial{l_{2}(\hat{\beta},\theta)}}{\partial{\lambda_{l}}}$ and $\frac{\partial{l_{2}(\hat{\beta},\theta)}}{\partial{\lambda_{l}}\partial{\lambda_{m}}}$, where l,m=1,…,q. From (A.1),

Similarly we can calculate the elements in ∇l _2R (θ ₁) and ∇² l _2R (θ ₁), i.e., $\frac{\partial{l_{2R}(\hat{\beta},\theta)}}{\partial{\lambda_{l}}}$ and $\frac{\partial{l_{2R}(\hat{\beta},\theta)}}{\partial{\lambda_{l}}\partial{\lambda_{m}}}$, where l,m=1,…,q. From (A.2),

For the derivatives with respect to θ ₂, we have similar expressions. Denote the elements in ∇l ₂(θ ₂) by $\frac{\partial{l_{2}(\hat{\beta},\theta)}}{\partial{\gamma_{s_{1}r_{1}}}}$ where s ₁=2,…,q; r ₁=1,…,l ₁−1. And denote the elements in ∇² l ₂(θ ₂) by $\frac{\partial{l_{2}(\hat{\beta},\theta)}}{\partial{\gamma_{s_{1}r_{1}}}\partial{\gamma_{s_{2}r_{2}}}}$, where s ₂=2,…,q; r ₂=1,…,l ₂−1. From (A.1),

Similarly form (A.2), we have

Finally, we give the expression forms of $\frac{\partial{V_{k}}}{\partial{\lambda_{l}}}$ ($\frac{\partial{V_{k}}}{\partial{\lambda_{m}}}$), $\frac{\partial^{2}{V_{k}}}{\partial{\lambda_{l}}\partial{\lambda_{m}}}$ and $\frac{\partial{V_{k}}}{\partial{\gamma_{s_{1}r_{1}}}}$ ($\frac{\partial{V_{k}}}{\partial{\gamma_{s_{2}r_{2}}}}$), $\frac{\partial^{2}{V_{k}}}{\partial{\gamma_{s_{1}r_{1}}}\partial{\gamma_{s_{2}r_{2}}}}$ in the above equations. First we have

where $\frac{\partial{\varLambda}}{\partial{\lambda_{l}}}$ is a q×q matrix with 1 at the lth diagonal entry and 0 at all the other entries. Also we have

where $\frac{\partial{\varLambda}}{\partial{\lambda_{m}}}$ is a q×q matrix with 1 at the mth diagonal entry and 0 at all the other entries.

If l≠m, it is easy to see that $\frac{\partial^{2}{V_{k}}}{\partial{\lambda_{l}}\partial{\lambda_{m}}}$ becomes to a q×q matrix with 0 at all entries; if l=m, $\frac{\partial^{2}{V_{k}}}{\partial{\lambda_{l}}\partial{\lambda_{m}}}=2*Z_{k}U_{l}Z_{k}^{T}$, where U _l is a q×q matrix with $1+\varSigma_{i=1}^{l-1}\gamma_{li}^{2}$ at the lth diagonal entry and 0 at all the other entries.

The expression form of $\frac{\partial{V_{k}}}{\partial{\gamma_{s_{1}r_{1}}}}$ is

where $\frac{\partial{\varGamma}}{\partial{\gamma_{s_{1}r_{1}}}}$ is a q×q matrix with 1 at the s ₁th row and r ₁th column entry and 0 at all the other entries. Furthermore, we have

If r ₁≠r ₂, $\frac{\partial{\varGamma}}{\partial{\gamma_{s_{1}r_{1}}}}(\frac{\partial{\varGamma}}{\partial{\gamma_{s_{2}r_{2}}}})^{T}$ turns to be a zero matrix, hence $\frac{\partial^{2}{V_{k}}}{\partial{\gamma_{s_{1}r_{1}}}\partial{\gamma_{s_{2}r_{2}}}}$ also becomes a zero matrix; if r ₁=r ₂, $\frac{\partial{\varGamma}}{\partial{\gamma_{s_{1}r_{1}}}}(\frac{\partial{\varGamma}}{\partial{\gamma_{s_{2}r_{2}}}})^{T}$ turns into a matrix with 1 at s ₁th row and s ₂th column entry and 0 at all the other entries. At this time, if s ₁≠s ₂, $\frac{\partial^{2}{V_{k}}}{\partial{\gamma_{s_{1}r_{1}}}\partial{\gamma_{s_{2}r_{2}}}}$ become a zero matrix since Λ is a diagonal matrix; if s ₁=s ₂, $\frac{\partial^{2}{V_{k}}}{\partial{\gamma_{s_{1}r_{1}}}\partial{\gamma_{s_{2}r_{2}}}}=2*Z_{k}S_{s_{1}}Z_{k}^{T}$, where $S_{s_{1}}$ is matrix with $\lambda_{s_{1}}^{2}$ at the s ₁th diagonal entry and 0 at all the other entries.