Nonparametric classification of high dimensional observations

Abstract

We consider the nonparametric classification of high dimensional, low sample size (HDLSS) data where the classical discrimination methods break down due to the singularity of the sample covariance matrix. We present new dissimilarity indices, discuss their asymptotic properties in the HDLSS setting, use them in building powerful classifiers, and compare their behavior with existing methods. We illustrate the difficulties with the Euclidean nearest neighbor method and prove that dissimilarity-based classifiers produce misclassification rates that tend to zero as $p\rightarrow \infty$. We present test-based classifiers in the HDLSS setting. A simulation study compares the misclassification rates of diagonal linear discriminant analysis with twelve other nonparametric classifiers. The methods are applied to microarray data for classification of prostate cancer.

Appendix

Proof

Lemma 3 Assuming (B1)–(B3) hold, if $\mathbf{z}$ is from $\pi _x$, then $\mathbb {I}_j^\delta (\mathbf{X}_i)\rightarrow 1$ and $\mathbb {I}_j^\delta (\mathbf{Y}_j)\rightarrow 0$ so that $T_1(NN_{\delta _0})\rightarrow k>0$ while $T_2(NN_{\delta _0})\rightarrow 0$ with probability 1 as $p\rightarrow \infty$. Hence, $T_1(NN_{\delta _0})>T_2(NN_{\delta _0})$ and $\mathbf{z}$ will be assigned to $\pi _x$ with probability 1 as $p\rightarrow \infty$. Similarly, if $\mathbf{z}$ is from $\pi _y$, then $\mathbb {I}_j^\delta (\mathbf{X}_i)\rightarrow 0$ and $\mathbb {I}_j^\delta (\mathbf{Y}_j)\rightarrow 1$ so that $T_1(NN_{\delta _0})\rightarrow 0$ while $T_2(NN_{\delta _0})\rightarrow k>0$ with probability 1 as $p\rightarrow \infty$. Hence, $T_1(NN_{\delta _0})<T_2(NN_{\delta _0})$ and $\mathbf{z}$ will be assigned to $\pi _x$ with probability 1 as $p\rightarrow \infty$. Replacing $\delta _0$ with $\rho _0$ in the above argument establishes that as $p\rightarrow \infty$, the NN method with index $\rho _0$ has zero misclassification rate as $p\rightarrow \infty$. $\square$

Proof

Lemma 4 Assuming (B1)–(B3) hold, and using comment 1, if $\mathbf{z}$ is from $\pi _x$, then $T_1(\delta _0)\rightarrow 0$ while $T_2(\delta _0)\rightarrow m\tilde{\delta }_0({\mathbb F},{\mathbb G})>0$ with probability 1 as $p\rightarrow \infty$. Hence, $T_1(\delta _0)<T_2(\delta _0)$ and $\mathbf{z}$ will be assigned to $\pi _x$ with probability 1 as $p\rightarrow \infty$ If $\mathbf{z}$ is from $\pi _y$, then $T_1(\delta _0)\rightarrow m\tilde{\delta }_0({\mathbb F},{\mathbb G})>0$ while $T_2(\delta _0)\rightarrow 0$ with probability 1 as $p\rightarrow \infty$. Hence, $T_1(\delta _0)>T_2(\delta _0)$ and $\mathbf{z}$ will be assigned to $\pi _y$ with probability 1 as $p\rightarrow \infty$.

Comment 2 explains the behavior of $T_1(\rho _0)$. If $\mathbf{z}$ is from $\pi _x$, then $T_1(\rho _0)\rightarrow 0$ while $T_2(\rho _0)\rightarrow m\tilde{\rho }_0({\mathbb F},{\mathbb G})>0$ with probability 1 as $p\rightarrow \infty$. Hence, $T_1(\rho _0)<T_2(\rho _0)$ and $\mathbf{z}$ will be assigned to $\pi _x$ with probability 1 as $p\rightarrow \infty$ If the new observation $\mathbf{z}$ is from $\pi _y$, then $T_1(\rho _0)\rightarrow m\tilde{\rho }_0({\mathbb F},{\mathbb G})>0$ while $T_2(\rho _0)\rightarrow 0$ with probability 1 as $p\rightarrow \infty$. Hence, $T_1(\rho _0)>T_2(\rho _0)$ and $\mathbf{z}$ will be assigned to $\pi _y$ with probability 1 as $p\rightarrow \infty$. $\square$

Proof

Lemma 1 We obtain $T_1(BF_\delta )=2\bar{\delta }_{(x\cup z)y}-\bar{\delta }_{x\cup z}-\bar{\delta }_{y}$. and $T_2({BF}_\delta )=2\bar{\delta }_{x(y\cup z)}-\bar{\delta }_{x}-\bar{\delta }_{y\cup z}$. Assuming (B1)–(B3) hold, if $\mathbf{z}$ is from $\pi _x$ and assigned to $\pi _x$, then there are $m+1$ observations in $x\cup z$ and n observations in the $\mathbf{Y}$ sample. Hence, using comment 2, $\bar{\delta }_{x\cup z}\rightarrow 0$, $\bar{\delta }_{(y)}\rightarrow 0$, and $\bar{\delta }_{(x\cup z)y}\rightarrow \tilde{\delta }_0({\mathbb F},{\mathbb G})$ as $p\rightarrow \infty$. Hence, $T_1({BF}_\delta )$ converges to $2\tilde{\delta }_0({\mathbb F},{\mathbb G})>0$ in probability as $p\rightarrow \infty$. If the new observation $\mathbf{z}$ is from $\pi _x$ and assigned to $\pi _y$, then there are m observations in $\mathbf{X}$ sample and $n+1$ observations in the $y\cup z$ sample. Hence, $\delta _{x}\rightarrow 0$, $\bar{\delta }_{y\cup z}\rightarrow \frac{2}{n+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$, and $\bar{\delta }_{x(y\cup z)}\rightarrow \frac{n}{n+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$. Hence, $T_2({BF}_\delta )$ converges to $\frac{2(n-1)}{n+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$ in probability as $p\rightarrow \infty$. Since $T_1(BF_{\delta _0})>T_2(BF_{\delta _0})$, $\mathbf{z}$ will be assigned to $\pi _x$ with probability 1 as $p\rightarrow \infty$.

Assuming (B1)–(B3) hold, if $\mathbf{z}$ is from $\pi _y$ and assigned to $\pi _x$, then there are $m+1$ observations in $x\cup z$ and n observations in the $\mathbf{Y}$ sample. Hence, $\bar{\delta }_{x\cup z}\rightarrow \frac{2}{m+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$, $\bar{\delta }_{(y)}\rightarrow 0$, and $\bar{\delta }_{(x\cup z)y}\rightarrow \frac{m}{m+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$ as $p\rightarrow \infty$. Now, $T_1({BF}_\delta )$ converges to $\frac{2(m-1)}{m+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})>0$ in probability as $p\rightarrow \infty$. If $\mathbf{z}$ is from $\pi _y$ and assigned to $\pi _y$, then there are m observations in $\mathbf{X}$ sample and $n+1$ observations in the $y\cup z$ sample. Hence, $\delta _{x}\rightarrow 0$, $\bar{\delta }_{y\cup z}\rightarrow 0$, and $\bar{\delta }_{x(y\cup z)}\rightarrow \frac{n}{n+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$. Hence, $T_2({BF}_\delta )$ converges to $2\tilde{\delta }_0({\mathbb F},{\mathbb G})$ in probability as $p\rightarrow \infty$. Since $T_1(BF_{\delta _0})<T_2(BF_{\delta _0})$, $\mathbf{z}$ will be assigned to $\pi _y$ with probability 1 as $p\rightarrow \infty$. $\square$

Proof

Lemma 2 We obtain $T_1({BG}_\delta )=(\bar{\delta }_{(x\cup z) y}- \bar{\delta }_{(x\cup z)})^2+(\bar{\delta }_{(x\cup z) y}- \bar{\delta }_{(y)})^2$ and $T_2({BG}_\delta )=(\bar{\delta }_{x (y\cup z)}- \bar{\delta }_{(y\cup z)})^2+(\bar{\delta }_{x(y\cup z)}- \bar{\delta }_{(x)})^2$. Assuming (B1)–(B3) hold, if $\mathbf{z}$ is from $\pi _x$ and assigned to $\pi _x$, then there are $m+1$ observations in $x\cup z$ and n observations in the $\mathbf{Y}$ sample. Hence, $\bar{\delta }_{x\cup z}\rightarrow 0$, $\bar{\delta }_{(y)}\rightarrow 0$, and $\bar{\delta }_{(x\cup z)y}\rightarrow \tilde{\delta }_0({\mathbb F},{\mathbb G})$ as $p\rightarrow \infty$. Hence, using comment 2, $T_1({BF}_\delta )$ converges to $2\tilde{\delta }^2_0({\mathbb F},{\mathbb G})>0$ in probability as $p\rightarrow \infty$. If $\mathbf{z}$ is from $\pi _x$ and assigned to $\pi _y$, then there are m observations in $\mathbf{X}$ sample and $n+1$ observations in the $y\cup z$ sample. Hence, $\delta _{x}\rightarrow 0$, $\bar{\delta }_{y\cup z}\rightarrow \frac{2}{n+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$, and $\bar{\delta }_{x(y\cup z)}\rightarrow \frac{n}{n+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$. Hence, $T_2({BF}_\delta )$ converges to $\frac{(n-2)^2+n^2}{(n+1)^2}\tilde{\delta }^2_0({\mathbb F},{\mathbb G})$ in probability as $p\rightarrow \infty$. Since $T_1(BG_{\delta _0})>T_2(BG_{\delta _0})$, $\mathbf{z}$ will be assigned to $\pi _x$ with probability 1 as $p\rightarrow \infty$.

Assuming (B1)–(B3) hold, if $\mathbf{z}$ is from $\pi _y$ and assigned to $\pi _x$, then there are $m+1$ observations in $x\cup z$ and n observations in the $\mathbf{Y}$ sample. Hence, using comment 1, $\bar{\delta }_{x\cup z}\rightarrow \frac{2}{m+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$, $\bar{\delta }_{(y)}\rightarrow 0$, and $\bar{\delta }_{(x\cup z)y}\rightarrow \frac{m}{m+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$ as $p\rightarrow \infty$. Hence, $T_1({BF}_\delta )$ converges to $\frac{(m-2)^2+m^2}{(m+1)^2}\tilde{\delta }^2_0({\mathbb F},{\mathbb G})>0$ in probability as $p\rightarrow \infty$. If the new observation $\mathbf{z}$ is from $\pi _y$ and assigned to $\pi _y$, then there are m observations in $\mathbf{X}$ sample and $n+1$ observations in the $y\cup z$ sample. Hence, $\delta _{x}\rightarrow 0$, $\bar{\delta }_{y\cup z}\rightarrow 0$, and $\bar{\delta }_{x(y\cup z)}\rightarrow \frac{m}{m+1}\tilde{\delta }_0({\mathbb F},{\mathbb G})$. Hence, $T_2({BF}_\delta )$ converges to $\frac{2m^2}{(m+1)^2}\tilde{\delta }^2_0({\mathbb F},{\mathbb G})$ in probability as $p\rightarrow \infty$. Since $T_1(BG_{\delta _0})<T_2(BG_{\delta _0})$, $\mathbf{z}$ will be assigned to $\pi _y$ with probability 1 as $p\rightarrow \infty$. $\square$