Robust estimation and variable selection in heteroscedastic regression model using least favorable distribution

Abstract

The assumption of equal variances is not always appropriate and different approaches for modelling variance heterogeneity have been widely studied in the literature. One of these approaches is joint location and scale model defined with the idea that both the location and the scale depend on explanatory variables through parametric linear models. Because the joint location and scale model includes two models, it does not deal well with a large number of irrelevant variables. Therefore, determining the variables that are important for the location and the scale is as important as estimating the parameters of these models. From this point of view, a combine robust estimation and variable selection method is proposed to simultaneously estimate the parameters and select the important variables. This is done using the least favorable distribution and least absolute shrinkage and selection operator method. Under appropriate conditions, we study the consistency, asymptotic distribution and the sparsity property of the proposed robust estimator. Simulation studies and a real data example are provided to demonstrate the advantages of the proposed method over existing methods in literature.

Appendix

Proof of Theorem 1

First, we have to show that ${Z}_{n}\left({\varvec{\theta}}\right)$ converges uniformly in probability to $Z\left({\varvec{\theta}}\right)$ given in (26), then ${\widehat{{\varvec{\theta}}}}_{n}$ is uniformly bounded in probability.

At first, we show that

$$\underset{{\varvec{\theta}}\in\Theta }{\mathrm{sup}}\left|{Z}_{n}\left({\varvec{\theta}}\right)-Z\left({\varvec{\theta}}\right)\right|\to 0$$

in probability. Since the third term in ${Z}_{n}\left({\varvec{\theta}}\right)$ is not stochastic and the parameter space is compact (A4), it is sufficient to show that $\frac{1}{2n}\sum_{i=1}^{n}{{\varvec{z}}}_{{\varvec{i}}}^{T}{\varvec{\gamma}}+\frac{1}{n}\sum_{i=1}^{n}\rho \left(\frac{{y}_{i}-{{\varvec{x}}}_{{\varvec{i}}}^{T}{\varvec{\upbeta}}}{{e}^{{{\varvec{z}}}_{{\varvec{i}}}^{T}{\varvec{\gamma}}/2}}\right)$ converges uniformly in probability to ${l}\left({\varvec{\theta}}\right)$ to show that ${Z}_{n}\left({\varvec{\theta}}\right)$ converges uniformly in probability to $Z\left({\varvec{\theta}}\right)$ (Arslan 2016).

$\rho$ function given in (9) is continuous (A4). Furthermore $\mathrm{sup}\rho \left(r;{\varvec{\theta}}\right)<\infty$, $\rho \left(t\right)\le \underset{\theta \in\Theta }{\mathrm{sup}}\rho \left(r;{\varvec{\theta}}\right)$ and $E\left[\underset{{\varvec{\theta}}\in\Theta }{\mathrm{sup}}\rho \left(r;{\varvec{\theta}}\right)\right]<\infty$ where $r=\frac{y-{{\varvec{x}}}^{T}{\varvec{\upbeta}}}{{e}^{{{\varvec{z}}}^{{\varvec{T}}}{\varvec{\gamma}}/2}}$. Thus, we have that $E\left[\rho \left(r;{\varvec{\theta}}\right)\right]$ is continuous and

$$\underset{\theta \in\Theta }{\mathrm{sup}}\left|\frac{1}{2n}\sum_{i=1}^{n}{{\varvec{z}}}_{{\varvec{i}}}^{T}{\varvec{\gamma}}+\frac{1}{n}\sum_{i=1}^{n}\rho \left(\frac{{y}_{i}-{{\varvec{x}}}_{{\varvec{i}}}^{T}{\varvec{\upbeta}}}{{e}^{{{\varvec{z}}}_{{\varvec{i}}}^{T}{\varvec{\gamma}}/2}}\right)-{l}\left({\varvec{\theta}}\right)\right|\to 0$$

in probability (Newey and McFadden 1994). This result combined with ${\lambda }_{n}/n\to 0$ implies that

$$\underset{\theta \in\Theta }{\mathrm{sup}}\left|{Z}_{n}\left({\varvec{\theta}}\right)-Z\left({\varvec{\theta}}\right)\right|\to 0$$

in probability. Further, since

$${Z}_{n}\left({\varvec{\theta}}\right)\ge \frac{1}{2n}\sum_{i=1}^{n}{{\varvec{z}}}_{{\varvec{i}}}^{T}{\varvec{\gamma}}+\frac{1}{n}\sum_{i=1}^{n}\rho \left(\frac{{y}_{i}-{{\varvec{x}}}_{{\varvec{i}}}^{T}{\varvec{\upbeta}}}{{e}^{{{\varvec{z}}}_{{\varvec{i}}}^{T}{\varvec{\gamma}}/2}}\right)$$

we have

$$\underset{{\varvec{\theta}}}{\mathrm{argmin}}\left(\frac{1}{2n}\sum_{i=1}^{n}{{\varvec{z}}}_{{\varvec{i}}}^{T}{\varvec{\gamma}}+\frac{1}{n}\sum_{i=1}^{n}\rho \left(\frac{{y}_{i}-{{\varvec{x}}}_{{\varvec{i}}}^{T}{\varvec{\upbeta}}}{{e}^{{{\varvec{z}}}_{{\varvec{i}}}^{T}{\varvec{\gamma}}/2}}\right)\right)={\widehat{{\varvec{\theta}}}}_{ML}={O}_{p}\left(1\right).$$

Then it follows that

$$\underset{{\varvec{\theta}}}{\mathrm{argmin}}{Z}_{n}\left({\varvec{\theta}}\right)={\widehat{{\varvec{\theta}}}}_{n}={O}_{p}\left(1\right).$$

Combining these results (the convergence in probability of ${Z}_{n}$ and ${\widehat{{\varvec{\theta}}}}_{n}$ is uniformly bounded) we obtain

$${\widehat{{\varvec{\theta}}}}_{n}=\underset{{\varvec{\theta}}}{\mathrm{argmin}}{Z}_{n}\left({\varvec{\theta}}\right)\to \underset{{\varvec{\theta}}}{\mathrm{argmin}}Z\left({\varvec{\theta}}\right).$$

Moreover, when ${\lambda }_{n}/n\to 0$ as $n\to \infty$, ${Z}_{n}\left({\varvec{\theta}}\right)$ converges uniformly in probability to ${l}\left({\varvec{\theta}}\right)$ and, since ${l}\left({\varvec{\theta}}\right)$ has a unique minimum at ${{\varvec{\theta}}}_{0}$ (A2) we get ${\widehat{{\varvec{\theta}}}}_{n}\to {{\varvec{\theta}}}_{0}$ in probability which confirms the consistency of ${\widehat{{\varvec{\theta}}}}_{n}.$

Proof of Theorem 2

Let us define ${Q}_{n}\left({\varvec{u}}\right)={\mathcal{L}}_{n}\left({{\varvec{\theta}}}_{0}+{n}^{-\frac{1}{2}}{\varvec{u}}\right)-{\mathcal{L}}_{n}\left({{\varvec{\theta}}}_{0}\right)$ with ${\varvec{u}}\in {\mathbb{R}}^{s}$. Obviously, ${Q}_{n}\left({\varvec{u}}\right)$ is minimized at ${\widehat{{\varvec{u}}}}_{n}=\sqrt{n}\left({\widehat{{\varvec{\theta}}}}_{n}-{{\varvec{\theta}}}_{0}\right)$ because ${\widehat{{\varvec{\theta}}}}_{n}$ minimizes ${\mathcal{L}}_{n}\left({\varvec{\theta}}\right)$. First, we need to show that

$${Q}_{n}\left({\varvec{u}}\right)\stackrel{D}{\to }Q\left({\varvec{u}}\right).$$

${Q}_{n}\left({\varvec{u}}\right)$ can be rewritten as

$$ \begin{aligned} Q_{n} \left( {\varvec{u}} \right) & = h\left( {{\varvec{\theta}}_{0} + n^{{ - \frac{1}{2}}} {\varvec{u}}} \right) - h\left( {{\varvec{\theta}}_{0} } \right) + \lambda_{n} \mathop \sum \limits_{j = 1}^{s} \left| {\theta_{0j} + n^{{ - \frac{1}{2}}} {\varvec{u}}} \right| - \lambda_{n} \mathop \sum \limits_{j = 1}^{s} \left| {\theta_{0j} } \right| \\ & = \left[ {h\left( {{\varvec{\theta}}_{0} + n^{{ - \frac{1}{2}}} {\varvec{u}}} \right) - h\left( {{\varvec{\theta}}_{0} } \right)} \right] + \lambda_{n} \mathop \sum \limits_{j = 1}^{{s_{1} }} \left( {\left| {\theta_{0j} + n^{ - 1/2} {\varvec{u}}} \right| - \left| {\theta_{0j} } \right|} \right) \\ & \quad + \lambda_{n} \mathop \sum \limits_{{j = s_{1} + 1}}^{{s_{2} }} \left( {\left| {\theta_{0j} + n^{ - 1/2} {\varvec{u}}} \right| - \left| {\theta_{0j} } \right|} \right). \\ \end{aligned} $$

For the first part of above equation, using Taylor series expansion around ${\varvec{u}}=0$, we get

$$h\left({{\varvec{\theta}}}_{0}+{n}^{-\frac{1}{2}}{\varvec{u}}\right)-h\left({{\varvec{\theta}}}_{0}\right)=-{n}^{-\frac{1}{2}}{{\varvec{u}}}^{T} {h}^{^{\prime}}\left({{\varvec{\theta}}}_{0}\right)+\frac{1}{2}{n}^{-1}{{\varvec{u}}}^{T} {h}^{{^{\prime}}{^{\prime}}}\left({{\varvec{\theta}}}_{0}\right){\varvec{u}}.$$

Since $\frac{1}{\sqrt{n}}{h}^{^{\prime}}\left({{\varvec{\theta}}}_{0}\right)\stackrel{D}{\to }{\varvec{W}}$ with ${\varvec{W}}\sim {N}_{s}\left(0,A\left({h}^{^{\prime}}\left({{\varvec{\theta}}}_{0}\right)\right)\right)$ and $\frac{1}{n}{h}^{{^{\prime}}{^{\prime}}}\left({{\varvec{\theta}}}_{0}\right)\to B\left({h}^{^{\prime}}\left({{\varvec{\theta}}}_{0}\right)\right)$ where $A\left({h}^{^{\prime}}\left({{\varvec{\theta}}}_{0}\right)\right)=E\left[{\left({h}^{^{\prime}}\left({{\varvec{\theta}}}_{0}\right)\right)}^{2}\right]$ and $B\left({h}^{^{\prime}}\left({{\varvec{\theta}}}_{0}\right)\right)=E\left[{h}^{{^{\prime}}{^{\prime}}}\left({{\varvec{\theta}}}_{0}\right)\right]$, we obtain

$$h\left({{\varvec{\theta}}}_{0}+{n}^{-\frac{1}{2}}{\varvec{u}}\right)-h\left({{\varvec{\theta}}}_{0}\right) \stackrel{D}{\to } -{{\varvec{u}}}^{T} {\varvec{W}}+\frac{1}{2}{{\varvec{u}}}^{T} \left(B\left({h}^{^{\prime}}\left({{\varvec{\theta}}}_{0}\right)\right)\right){\varvec{u}}$$

(Arslan 2016).

Similar to Knight and Fu (2000) and Arslan (2016), we have

$$ \begin{aligned} & \lambda_{n} \mathop \sum \limits_{j = 1}^{s} \left( {\left| {\theta_{0j} + n^{{ - \frac{1}{2}}} {\varvec{u}}} \right| - \left| {\theta_{0j} } \right|} \right) = \frac{{\lambda_{n} }}{\sqrt n }\mathop \sum \limits_{j = 1}^{s} \left( {u_{j} sgn\left( {\theta_{0j} } \right)\left( {\theta_{0j} \ne 0} \right) + \left| {u_{j} } \right|I\left( {\theta_{0j} = 0} \right)} \right) \\ & \quad \to \frac{{\lambda_{0} }}{\sqrt n }\mathop \sum \limits_{j = 1}^{s} \left( {u_{j} sgn\left( {\theta_{0j} } \right)\left( {\theta_{0j} \ne 0} \right) + \left| {u_{j} } \right|I\left( {\theta_{0j} = 0} \right)} \right) \\ \end{aligned} $$

as $n\to \infty$. Then, we obtain

$${Q}_{n}\left({\varvec{u}}\right)\stackrel{D}{\to }Q\left({\varvec{u}}\right)$$

as $n\to \infty$. Since $Q\left({\varvec{u}}\right)$ has a unique minimum and ${Q}_{n}\left({\varvec{u}}\right)$ can be approximated by a convex function, we finally have

$$\sqrt{n}\left({\widehat{{\varvec{\theta}}}}_{n}-{{\varvec{\theta}}}_{0}\right)=\underset{{\varvec{u}}}{\mathrm{argmin}}{Q}_{n}\left({\varvec{u}}\right)\stackrel{D}{\to }\underset{{\varvec{u}}}{\mathrm{argmin}}Q({\varvec{u}}).$$

Proof of Theorem 3

First, we prove that for any given ${{\varvec{\theta}}}^{({s}_{1})}$ satisfying ${{\varvec{\theta}}}^{({s}_{1})}-{{\varvec{\theta}}}_{0}^{({s}_{1})}=O\left({n}^{-1/2}\right)$ and any constant $c > 0$, we have

$${\mathcal{L}}_{n}\left\{{\left({\left({{\varvec{\theta}}}^{({s}_{1})}\right)}^{T},{0}^{T}\right)}^{T}\right\}=\underset{\Vert {{\varvec{\theta}}}^{({s}_{2})}\Vert \le c{n}^{-1/2}}{\mathrm{min}}{\mathcal{L}}_{n}\left\{{\left({\left({{\varvec{\theta}}}^{({s}_{1})}\right)}^{T},{\left({{\varvec{\theta}}}^{({s}_{2})}\right)}^{T}\right)}^{T}\right\}.$$

Simple calculations lead to the following expression of the derivative of $Q\left({\varvec{\theta}}\right)$.

$$\frac{\partial {\mathcal{L}}_{n}\left({\varvec{\theta}}\right)}{\partial {\theta }_{j}}=\frac{\partial h\left({\varvec{\theta}}\right)}{\partial {\theta }_{j}}+{\lambda }_{n}sgn\left({\theta }_{j}\right)$$

Then applying the Taylor’s expansion, for any ${\theta }_{j}$ $\left(j={s}_{1}+1,{s}_{1}+2,\dots ,s\right)$ we obtain

$$\frac{\partial {\mathcal{L}}_{n}\left({\varvec{\theta}}\right)}{\partial {\theta }_{j}}=\frac{\partial h\left({{\varvec{\theta}}}_{0}\right)}{\partial {\theta }_{j}}+\sum_{k=1}^{s}\frac{{\partial }^{2}h\left({{\varvec{\theta}}}^{*}\right)}{\partial {\theta }_{j}\partial {\theta }_{k}}\left({\theta }_{k}-{\theta }_{0k}\right)+{\lambda }_{n}sgn\left({\theta }_{j}\right)$$

where ${{\varvec{\theta}}}^{*}$ is between ${\varvec{\theta}}$ and ${{\varvec{\theta}}}_{0}$. On the other hand, we know that (Fan and Li 2001)

$$ \begin{aligned} & \frac{1}{n}\frac{{\partial h\left( {{\varvec{\theta}}_{0} } \right)}}{{\partial \theta_{j} }} = O_{p} \left( {n^{ - 1/2} } \right), \\ & \frac{1}{n}\left\{ {\frac{{\partial^{2} h\left( {{\varvec{\theta}}_{0} } \right)}}{{\partial \theta_{j} \partial \theta_{k} }}} \right\} - E\left[ {\frac{{\partial^{2} h\left( {{\varvec{\theta}}_{0} } \right)}}{{\partial \theta_{j} \partial \theta_{k} }}} \right] = O_{p} \left( 1 \right). \\ \end{aligned} $$

According to Theorem 1, it is clear that $\Vert {{\widehat{{\varvec{\theta}}}}_{n}-{\varvec{\theta}}}_{0}\Vert ={O}_{p}\left({n}^{-1/2}\right).$ Then, we obtain

$$\frac{\partial {\mathcal{L}}_{n}\left({\varvec{\theta}}\right)}{\partial {\theta }_{j}}={\lambda }_{n}\left\{-{\lambda }_{n}^{-1}{p}_{{\lambda }_{n}}^{^{\prime}}\left(\left|{\theta }_{j}\right|\right)+{O}_{p}\left({{\lambda }_{n}^{-1}n}^{-\frac{1}{2}}\right)\right\}={\lambda }_{n}\left\{-sgn\left({\theta }_{j}\right)+{O}_{p}\left({{\lambda }_{n}^{-1}n}^{-\frac{1}{2}}\right)\right\}.$$

While ${{\lambda }_{n}^{-1}n}^{-\frac{1}{2}}\to 0$ as $n\to \infty ,$ the sign of the derivative is completely determined by that of ${\theta }_{j}.$

Namely, we can ensure that

$$\left\{\begin{array}{ll}\frac{\partial {\mathcal{L}}_{n}\left({\varvec{\theta}}\right)}{\partial {\theta }_{j}}<0,&\quad for\, 0<{\theta }_{j}<c{n}^{-1/2}\\ \frac{\partial {\mathcal{L}}_{n}\left({\varvec{\theta}}\right)}{\partial {\theta }_{j}}>0,&\quad for\, -c{n}^{-1/2}<{\theta }_{j}<0.\end{array}\right.$$

Hence, with probability tending to 1, ${\mathcal{L}}_{n}\left({\varvec{\theta}}\right)$ achieve its minimum at ${\varvec{\theta}}={\left({\left({{\varvec{\theta}}}^{({s}_{1})}\right)}^{T},{0}^{T}\right)}^{T}.$ This completes the proof of Theorem 3.