Shrinkage estimation of varying covariate effects based on quantile regression

Abstract

Varying covariate effects often manifest meaningful heterogeneity in covariate-response associations. In this paper, we adopt a quantile regression model that assumes linearity at a continuous range of quantile levels as a tool to explore such data dynamics. The consideration of potential non-constancy of covariate effects necessitates a new perspective for variable selection, which, under the assumed quantile regression model, is to retain variables that have effects on all quantiles of interest as well as those that influence only part of quantiles considered. Current work on l ₁-penalized quantile regression either does not concern varying covariate effects or may not produce consistent variable selection in the presence of covariates with partial effects, a practical scenario of interest. In this work, we propose a shrinkage approach by adopting a novel uniform adaptive LASSO penalty. The new approach enjoys easy implementation without requiring smoothing. Moreover, it can consistently identify the true model (uniformly across quantiles) and achieve the oracle estimation efficiency. We further extend the proposed shrinkage method to the case where responses are subject to random right censoring. Numerical studies confirm the theoretical results and support the utility of our proposals.

Appendices

Appendix A: Proof of Theorem 1

Define B _n,τ (C)={β: β=β ₀(τ)+(n ^−1/2 ℓ ₂ n)u,∥u∥≤C}, and let ∂B _n,τ (C) denote the boundary set of B _n,τ (C). Since $W_{n, \lambda_{n}}({\boldsymbol {\beta }}; \tau)$ is convex in β for all τ∈Δ, it is sufficient to show that for any ϵ>0, there exists C ₀>0 and N ₀>0 such that for n≥N ₀,

$$\begin{aligned} &{\operatorname {pr}\Bigl(\inf_{\tau\in \varDelta }\Bigl[\inf_{{\boldsymbol {\beta }}\in\partial B_{n, \tau}(C_0)} W_{n, \lambda_n}({\boldsymbol {\beta }}; \tau)-W_{n, \lambda_n}\bigl\{{\boldsymbol {\beta }}_0( \tau); \tau\bigr\}\Bigr]>0\Bigr) } \\ &{\quad \geq1-\epsilon. } \end{aligned}$$

(2)

To show (2), first note that

Write and u(τ;β)=β−β ₀(τ), and let $\mathcal {D}$ and $\mathcal {D}_{i}$ respectively denote operators such that $\mathcal {D}(U)=U-E(U)$ and $\mathcal {D}_{i}(U)=U-E(U|\boldsymbol {Z}_{i})$ for a random variable U. We can further decomposed the term I as I=III+IV, where

and

For the term IV ₁, we can show that the function class, , is a Donsker class (page 81 in Van der Vaart and Wellner (2000)), where $\mathcal{A}(C_{0})=\{\boldsymbol {b}\in R^{p}:\ \inf_{\tau\in \varDelta }\|\boldsymbol {b}-{\boldsymbol {\beta }}_{0}(\tau)\|\leq C_{0}\}$. Given the uniform boundedness of the functional class $\mathcal{F}$ and since $\mathcal{A}(C_{0})$ covers $\mathcal{B}_{n,\tau}(C_{0})$ for all τ∈Δ, we can apply the functional law of the iterated logarithm (LIL) (Goodman et al. 1981) to $n^{-1}\sum_{i=1}^{n}$ , and get

$$\sup_{\tau\in \varDelta , {\boldsymbol {\beta }}\in\partial B_{n, \tau }(C_0)}|\mathit{IV}_1|\leq O_p \bigl(n^{-1/2}\ell_2 n\bigr) \bigl(C_0 n^{-1/2}\ell_2 n\bigr). $$

Similarly, we can show that, for j=2,…,8,

$$\sup_{\tau\in \varDelta , {\boldsymbol {\beta }}\in\partial B_{n, \tau}(C_0)}\|\mathit{IV}_j\|\leq O_p \bigl(n^{-1/2}\ell_2 n\bigr) \bigl(C_0 n^{-1/2}\ell_2 n\bigr). $$

Therefore, we have

$$ \sup_{\tau\in \varDelta , {\boldsymbol {\beta }}\in\partial B_{n, \tau}(C_0)}|\mathit{IV}|\leq O_p \bigl(n^{-1/2}\ell_2 n\bigr) \bigl(C_0 n^{-1/2}\ell_2 n\bigr). $$

(3)

For the term III, note that, under condition C2 (i), inf _τ∈Δ,z f _τ (0|z)>0. By the definition of B _n,τ (C ₀), there exists N ₁>0 such that for n≥N ₁,

This, coupled with condition C2 (ii), implies that for n>N ₁, f _τ (x|z)>inf _τ∈Δ,z f _τ (0|z)/2 for any with β∈∂B _n,τ (C ₀). Let δ ₂≡inf _τ,z f _τ (0|z)/2 and , where ${\rm eig}\min(\cdot)$ denotes the minimum eigenvalue of a matrix. Let E _Z (⋅) denote the expectation with regard to Z. We get, for any τ∈Δ and β∈∂B _n,τ (C ₀),

Since a similar result can be shown for the second term in III, it follows that

$$ \inf_{{\boldsymbol {\beta }}\in\partial B_{n, \tau}(C_0), \tau\in \varDelta } \mathit{III}\geq C_0^2 O_p\bigl((\ell_2 n)^2/n\bigr). $$

(4)

For the term II, it is easy to see that

$$\begin{aligned} \mathit{II}\geq -\frac{\lambda_n}{n}\sum_{j=2}^s\bigl| \beta^{(j)}-\beta_0^{(j)}(\tau )\bigr|w_{n,j}( \tau). \end{aligned}$$

By the uniform consistency of $\tilde{\beta}_{n}(\tau)$, for 2≤j≤s, $\sup_{\tau\in \varDelta } |w_{n,j}(\tau)|=(\sup_{\tau\in \varDelta }|\tilde{\beta}_{n}^{(j)}(\tau )|)^{-1}=O_{p}(1)$. Therefore, with (n ^1/2 ℓ ₂ n)⁻¹ λ _n =O(1),

$$ \inf_{{\boldsymbol {\beta }}\in\partial B_{n, \tau}(C_0), \tau\in \varDelta } \mathit{II}\geq-C_0 \cdot O_p\bigl((\ell_2 n)^2/n\bigr). $$

(5)

Based on equations (3), (4), and (5), it follows that

$$\begin{aligned} &{\inf_{\tau\in \varDelta , {\boldsymbol {\beta }}\in\partial B_{n, \tau}(C_0)}\bigl[n^{-1}W_{n, \lambda_n}({\boldsymbol {\beta }}; \tau)-n^{-1} W_{n, \lambda_n}\bigl\{{\boldsymbol {\beta }}_0(\tau); \tau \bigr\}\bigr]} \\ &{\quad \geq C_0^2\cdot O_p\bigl(( \ell_2 n)^2/n\bigr)-C_0\cdot O_p \bigl((\ell_2 n)^2/n\bigr).} \end{aligned}$$

Therefore, (2) holds if we choose C ₀ large enough. This completes the proof of Theorem 1.

Appendix B: Proof of Theorem 2

Define $\operatorname {sgn}(x)=I(x>0)-I(x<0)$, $U_{n, j}({\boldsymbol {\beta }}; \tau)= \frac {\partial W_{n, \lambda_{n}}({\boldsymbol {\beta }}; \tau)}{\partial\beta^{(j)}}$, and let . Define , , and .

First, by Theorem 1, for j=2,…,s, we have

$$\begin{aligned} &{\lim_{n\rightarrow\infty} \operatorname {pr}\Bigl(\sup _{\tau\in \varDelta }\bigl|\widehat{\beta}_{n, \lambda_n}^{\mathrm {US}(j)}(\tau)\bigr|=0 \Bigr) } \\ &{\quad \leq\lim_{n\rightarrow\infty} \operatorname {pr}\Bigl(\sup_{\tau\in \varDelta }\bigl| \widehat{\beta}_{n, \lambda_n}^{\mathrm {US}(j)}(\tau)-\beta_0^{(j)}( \tau)\bigr|\geq \sup_{\tau\in \varDelta }\bigr|\beta_0^{(j)}(\tau)\bigr| \Bigr) } \\ &{\quad =0.} \end{aligned}$$

(6)

Next, we note that given the uniform consistency of $\widehat{{\boldsymbol {\beta }}}_{n, \lambda_{n}}^{\mathrm {US}}(\tau)$ implied by Theorem 1, it can be shown by following the lines of Lemma 1 in Peng and Huang (2008) that

(7)

This implies, for j=s+1,…,p,

$$\begin{aligned} &{\sup_{\tau\in \varDelta }\bigl|E_{n, j}(\tau)\bigr| }\\ &{\quad \equiv\sup_{\tau\in \varDelta }\bigl|n^{-1/2} U_{n, j} \bigl\{\widehat{{\boldsymbol {\beta }}}_{n, \lambda_n}^{\mathrm {US}}(\tau); \tau\bigr\} -n^{-1/2} U_{n, j}\bigl\{{\boldsymbol {\beta }}_0(\tau); \tau\bigr\} } \\ &{\qquad{} -n^{1/2}\bigl[\mu_j\bigl\{\widehat{{\boldsymbol {\beta }}}_{n, \lambda_n}^{\mathrm {US}}(\tau)\bigr\}-\mu_j\bigl\{ {\boldsymbol {\beta }}_0(\tau)\bigr\}\bigr] } \\ &{\qquad{} -n^{-1/2}\lambda_n w_{n, j}(\tau)\operatorname {sgn}\bigl\{\widehat{\beta}_{n,\lambda_n}^{\mathrm {US}(j)}(\tau)\bigr\}\bigr|=o_p(1)} \end{aligned}$$

(8)

By the definition of $\widehat{{\boldsymbol {\beta }}}_{n, \lambda_{n}}^{\mathrm {US}}(\tau)$,

$$ \sup_{\tau\in \varDelta }\bigl|n^{-1/2} U_{n, j} \bigl\{\widehat{{\boldsymbol {\beta }}}_{n, \lambda_n}^{\mathrm {US}}(\tau); \tau\bigr \}\bigr|=o_p(1). $$

(9)

Applying Functional LIL to n ⁻¹ U _n,j {β ₀(τ);τ} gives

$$ \sup_{\tau\in \varDelta }\bigl|n^{-1/2}U_{n,j}\bigl \{{\boldsymbol {\beta }}_0(\tau); \tau\bigr\}\bigr|=O_p(\ell_2 n). $$

(10)

An application of Taylor expansion to $\mu_{j}\{\widehat{{\boldsymbol {\beta }}}_{n, \lambda_{n}}^{\mathrm {US}}(\tau)\}-\mu_{j}\{{\boldsymbol {\beta }}_{0}(\tau)\}$, coupled with Theorem 1 and the uniform boundedness of A _j (β), shows that

$$ \sup_{\tau\in \varDelta }\bigl|n^{1/2}\bigl[ \mu_j\bigl\{\widehat{{\boldsymbol {\beta }}}_{n, \lambda_n}^{\mathrm {US}}(\tau)\bigr \}-\mu_j\bigl\{{\boldsymbol {\beta }}_0(\tau)\bigr\} \bigr]\bigr|=O_p(\ell_2 n). $$

(11)

In addition, the proof of Theorem 1 can be used to justify that $\sup_{\tau\in \varDelta }|\tilde{\beta}_{n}^{(j)}(\tau )|=O_{p}(n^{-1/2}\ell_{2} n)$ and thus

$$ \lim_{n\rightarrow\infty} \biggl(\frac{1}{\ell_2 n} \biggr) \bigl(n^{-1/2}\lambda_n w_{n,j}(\tau) \bigr)=\infty. $$

(12)

By (9), (10), (11), and (12), with a fixed M>0,

$$\lim_{n\rightarrow\infty} \operatorname {pr}\Bigl(\sup_{\tau\in \varDelta }\bigl|E_{n,j}( \tau )\bigr|\leq M, \sup_{\tau\in \varDelta }\bigl|\widehat{{\boldsymbol {\beta }}}_{n, \lambda_n}^{\mathrm {US}(j)}( \tau)\bigr|\ne0\Bigr)=0. $$

This, coupled with (8), implies that, for j=s+1,…,p,

$$\begin{aligned} &{\lim_{n\rightarrow\infty} \operatorname {pr}\Bigl(\sup_{\tau\in \varDelta }\bigl| \widehat{{\boldsymbol {\beta }}}_{n, \lambda_n}^{\mathrm {US}(j)}(\tau)\bigr|\ne 0\Bigr) } \\ &{\quad \leq\lim_{n\rightarrow\infty}\Bigl\{\operatorname {pr}\Bigl(\sup_{\tau\in \varDelta }\bigl|E_{n,j}( \tau )\bigr|\leq M, \sup_{\tau\in \varDelta }\bigr|\widehat{{\boldsymbol {\beta }}}_{n, \lambda_n}^{\mathrm {US}(j)}( \tau)\bigr|\ne0\Bigr) } \\ &{\qquad{} +\operatorname {pr}\bigl(\bigl|E_{n,j}(\tau)\bigr|>M\bigr)\Bigr\}=0.} \end{aligned}$$

(13)

The proof of Theorem 2(i) is completed based on (6) and (13).

Let ${\buildrel a\over =}$ denote asymptotic equivalence in the sense that the difference converges to zero in probability uniformly in τ∈Δ. Define . By the result in Theorem 2(i), we have

$$n^{-1/2}\boldsymbol {U}_n(\bar{{\boldsymbol {\beta }}}_n; \tau)\,{\buildrel a\over =}\, n^{-1/2} \boldsymbol {U}_n\bigl(\widehat{{\boldsymbol {\beta }}}_{n, \lambda_n}^{\mathrm {US}}; \tau\bigr) $$

and hence $n^{-1/2}\boldsymbol {U}_{n}(\bar{{\boldsymbol {\beta }}}_{n}; \tau){\buildrel a\over =}0$. Using the result in (7) and applying Taylor expansion to $\mu_{j}\{\bar{{\boldsymbol {\beta }}}_{n}(\tau)\}-\mu_{j}\{{\boldsymbol {\beta }}_{0}(\tau)\}$, we get

(14)

where $\check{{\boldsymbol {\beta }}}(\tau)$ is on the line segment between β ₀(τ) and $\bar{{\boldsymbol {\beta }}}(\tau)$, and .

Since lim _n→∞ n ^−1/2 λ _n =0, sup _τ∈Δ |w _n,j (τ)|=O _p (1) for j=1,…,s, and $A\{\check{{\boldsymbol {\beta }}}(\tau)\}{\buildrel a\over =}A\{{\boldsymbol {\beta }}_{0}(\tau)\}$ given the uniform convergence of $\widehat{{\boldsymbol {\beta }}}_{n,\lambda_{n}}^{\mathrm {US}}(\tau)$ to β ₀(τ), it follows from (14) that

(15)

where A ₁₁(⋅) stands for the submatrix of A(⋅) formed by the first s rows and columns. An application of the Donsker theorem based on (15) thus shows that $n^{1/2}\{\widehat{{\boldsymbol {\beta }}}_{n,\lambda_{n}}^{\mathrm {US}(1:s)}(\tau)-{\boldsymbol {\beta }}_{0}^{(1:s)}(\tau)\}$ converges weakly to a mean zero Gaussian process with the covariance matrix,

Using similar steps, we can show that (15) still holds when $\widehat{{\boldsymbol {\beta }}}_{\rm oracle}(\tau)$ is in place of $\widehat{{\boldsymbol {\beta }}}_{n,\lambda_{n}}^{\mathrm {US}(1:s)}(\tau)$. This completes the proof of Theorem 2.