On Hodges’ superefficiency and merits of oracle property in model selection

Abstract

The oracle property of model selection procedures has attracted a large volume of favorable publications in the literature, but also faced criticisms of being ineffective and misleading in applications. Such criticisms, however, have appeared to be largely ignored by the majority of the popular statistical literature, despite their serious impact. In this paper, we present a new type of Hodges’ estimators that can easily produce model selection procedures with the oracle and some other desired properties, but can be readily seen to perform poorly in parts of the parameter spaces that are fixed and independent of sample sizes. Consequently, the merits of the oracle property for model selection as extensively advocated in the literature are questionable and possibly overstated. In particular, because the mathematics employed in this paper are at an elementary level, this finding leads to new discoveries on the merits of the oracle property and exposes some overlooked crucial facts on model selection procedures.

A Appendix: Proofs of the theorems

1.1 A.1 Proof of Theorem 1

Proof

Note first that $\hat{\theta }_{n}\overset{p}{\rightarrow }\theta$. For any $\theta \not =c$, the condition $a_{n}=o(1)$ implies that, for any $\varepsilon >0$,

$$\begin{aligned} \mathrm{Pr}_{\theta }(r_{n}\Vert \breve{\theta }_{n}(c)-\hat{\theta }_{n}\Vert >\varepsilon )&\le \mathrm{Pr}_{\theta }(\Vert \hat{\theta }_{n}-c\Vert \le a_{n})\le \mathrm{Pr}_{\theta }(\Vert \theta -c\Vert -\Vert \hat{\theta }_{n}-\theta \Vert \le a_{n})\\&=\mathrm{Pr}_{\theta }(\Vert \hat{\theta }_{n}-\theta \Vert \ge \Vert \theta -c\Vert -a_{n})\rightarrow 0\quad \hbox {as }n\rightarrow \infty . \end{aligned}$$

Thus, $r_{n}(\breve{\theta }_{n}(c)-\theta )=r_{n}(\breve{\theta }_{n}(c)-\hat{\theta }_{n})+r_{n}(\hat{\theta }_{n}-\theta )\overset{d}{\rightarrow }Z$. For $\theta =c$, thanks to $r_{n}a_{n}\rightarrow \infty$,

$$\begin{aligned} \mathrm{Pr}_{c}(r_{n}\Vert \breve{\theta }_{n}(c)-c\Vert>\varepsilon )\le \mathrm{Pr}_{c} (\breve{\theta }_{n}\not =c) =\mathrm{Pr}_{\theta _{0}}(r_{n}\Vert \hat{\theta }_{n}-c\Vert >r_{n}a_{n}) \rightarrow 0. \end{aligned}$$

(19)

This shows $r_{n}(\breve{\theta }_{n}(c)-c)\overset{p}{\rightarrow }0$. $\square$

1.2 A.2 Proof of Theorem 2

Proof

Because we are concerned with the asymptotic distribution of ${\check{\theta }}_{n, b}$ in this section, without loss of generality we can treat the easy case where V is known and ${\check{\theta }}_{n, b}$ is defined by

$$\begin{aligned} {\check{\theta }}_{n, b}= {\hat{\theta }}_{n,b}+{V}_{bb}^{-1}{V}_{b\bar{b}} ({\hat{\theta }}_{n,\bar{b}}-c_{{\bar{b}}}) \quad \hbox {and}\quad {\check{\theta }}_n(b)=({\check{\theta }}_{n,b}',c_{{\bar{b}}}')' \end{aligned}$$

(20)

with the convention ${\check{\theta }}_{n,\{1,2, \ldots ,d\}}={\check{\theta }}_{n}(\{1,2, \ldots ,d\}) ={\hat{\theta }}_{n}$.

The first assertion is obvious, so we here only prove (9). With the definition of $b(\theta )$ in (4), it is clear that $\theta _{b(\theta )}$ is the sub-vector $(\theta _j: \theta _j\ne c_j)$ of $\theta$ and $\theta _{\bar{b}(\theta )}=(c_j:j\in \bar{b}(\theta ))=c_{\bar{b}(\theta )}$. Note that, by (20), ${\check{\theta }}_{n,b(\theta )}={\hat{\theta }}_{n,b(\theta )}$ and hence ${\check{\theta }}_n(b(\theta ))=({\check{\theta }}_{n,b(\theta )},c_{{\bar{b}}(\theta )})$ are only pseudo-estimators that depend on the unknown parameters $\theta$. However,

$$\begin{aligned} r_n({\check{\theta }}_{n,b(\theta )}-\theta _{b(\theta )})&=V_{b(\theta ),b(\theta )}^{-1}(V_{b(\theta )b(\theta )}\quad V_{b(\theta ),\bar{b}(\theta )})r_n\left( \begin{array}{c} {\hat{\theta }}_{n,{b}(\theta )}-\theta _{b(\theta )}\\ {\hat{\theta }}_{n,\bar{b}(\theta )}-\theta _{\bar{b}(\theta )}\end{array}\right) \nonumber \\&\overset{d}{\rightarrow } V_{{b}(\theta ),{b}(\theta )}^{-1}(V_{{b}(\theta ){b}(\theta )}\quad V_{{b}(\theta ),\bar{b}(\theta )})Z =\check{Z}_{b(\theta )}, \end{aligned}$$

(21)

where the components of Z have been rearranged according to the order in $\theta$ and $\check{Z}_{b(\theta )}$ is derived from (7) by replacing b with $b(\theta )$.

For any $\theta$, by comparing (6) and (20),

$$\begin{aligned} \mathrm{Pr}_\theta \left( r_n\Vert {\tilde{\theta }}_n(c)-{\check{\theta }}_{n} (b(\theta ))\Vert>\varepsilon \right)&=\mathrm{Pr}_\theta \left( r_n\Vert {\tilde{\theta }} _n(c)-({\check{\theta }}_{b(\theta )}', c_{{\bar{b}}(\theta )}')'\Vert >\varepsilon \right) \\&\le \mathrm{Pr}_\theta \left( {\tilde{\theta }}_n(c)\ne ({\check{\theta }}_ {b(\theta )}',c_{{\bar{b}}(\theta )}')'\right) \\&\le \mathrm{Pr}_\theta (b_n(c)\ne b(\theta )). \end{aligned}$$

Since $\{b_n(c)\ne b(\theta )\}=\bigcup _{j\in b(\theta )}\{|{\hat{\theta }}_{nj}-c_j|\le a_{nj}\}\bigcup _{j\in {\bar{b}}(\theta )}\{|{\hat{\theta }}_{nj}-c_j|> a_{nj}\}$,

$$\begin{aligned} \mathrm{Pr}_\theta&\left( r_n\Vert {\tilde{\theta }}_n(c)-{\check{\theta }}_{n}(b(\theta )) \Vert>\varepsilon \right) \\&\le \mathrm{Pr}_\theta \left( \bigcup _{j\in b(\theta )}\{|{\hat{\theta }}_{nj}-c_j|\le a_{nj}\}\bigcup _{j\in {\bar{b}}(\theta )}\{|{\hat{\theta }}_{nj}-c_j|> a_{nj}\} \right) \\&\le \sum _{j\in b(\theta )}\mathrm{Pr}_\theta \{|{\hat{\theta }}_{nj}-c_j|\le a_{nj}\}+\sum _{j\in {\bar{b}}(\theta )} \mathrm{Pr}_\theta \{|{\hat{\theta }}_{nj}-c_j|> a_{nj}\} \\&\le \sum _{j\in b(\theta )}\mathrm{Pr}_\theta \{|{\hat{\theta }}_{nj}-\theta _j|\ge |\theta _j-c_j|-a_{nj}\} +\sum _{j\in {\bar{b}}(\theta )}\mathrm{Pr}_\theta \{r_n|{\hat{\theta }}_{nj}-c_j|> r_na_{nj}\}\\&\rightarrow 0 \quad \hbox {as }n\rightarrow \infty \end{aligned}$$

under the conditions on $a_n$. Thus, $r_n({\tilde{\theta }}_n(c)-{\check{\theta }}_{n}(b(\theta ))=o_p(1)$. Combining this with (21), we get

$$\begin{aligned} r_n({\tilde{\theta }}_n(c)-\theta )=r_n({\tilde{\theta }}_n(c) -{\check{\theta }}_{n}(b(\theta ))) +r_n({\check{\theta }}_{n}(b(\theta ))-\theta )\overset{d}{\rightarrow } \left( \begin{array}{c}\check{Z}_{b(\theta )}\\ 0\end{array}\right) \end{aligned}$$

(22)

under $\mathrm{Pr}_\theta$. Next examine the case $b(\theta )=\emptyset$. Analogous to (19), under $\mathrm{Pr}_c$ and the condition $r_n\min \limits _{1\le j\le d} a_{nj}\rightarrow \infty$,

$$\begin{aligned}&\mathrm{Pr}_{c}(r_{n}\Vert {\tilde{\theta }}_{n}(c)-c\Vert>\varepsilon )\le \mathrm{Pr}_{c} ({\tilde{\theta }}_{n}\not =c)\nonumber \\&\quad \le \sum _{j=1}^d\mathrm{Pr}_{c}(r_{n}|\hat{\theta }_{nj}-c|>r_{n}a_{nj}) \rightarrow 0 \quad \hbox {as }n\rightarrow \infty . \end{aligned}$$

(23)

Hence $r_{n}|{\tilde{\theta }}_{n}(c)-c| \overset{d}{\rightarrow }0$. The proof is then complete. $\square$