Asymptotic comparison of semi-supervised and supervised linear discriminant functions for heteroscedastic normal populations

Abstract

It has been reported that using unlabeled data together with labeled data to construct a discriminant function works successfully in practice. However, theoretical studies have implied that unlabeled data can sometimes adversely affect the performance of discriminant functions. Therefore, it is important to know what situations call for the use of unlabeled data. In this paper, asymptotic relative efficiency is presented as the measure for comparing analyses with and without unlabeled data under the heteroscedastic normality assumption. The linear discriminant function maximizing the area under the receiver operating characteristic curve is considered. Asymptotic relative efficiency is evaluated to investigate when and how unlabeled data contribute to improving discriminant performance under several conditions. The results show that asymptotic relative efficiency depends mainly on the heteroscedasticity of the covariance matrices and the stochastic structure of observing the labels of the cases.

Appendices

Appendix A: Asymptotic covariance matrix of $\hat{{\varvec{\vartheta }}}_\mathrm{C,Ind}$

In the appendices, we give the details of the asymptotic covariance matrices of estimators of ${\varvec{\vartheta }}$ based on four score functions. First, we derive the asymptotic covariance matrix of the estimator of ${\varvec{\theta }}$ based on $S_\mathrm{C,Ind}({\varvec{\theta }})$. This is obtained by calculating $\mathrm{E}\left[ \frac{\partial ^2}{\partial {\varvec{\theta }}\partial {\varvec{\theta }}'}S_\mathrm{C,Ind}({\varvec{\theta }})\right]$ and taking its inverse. Denote this matrix by $\bar{{\varvec{\Lambda }}}_\mathrm{C,Ind}$. Then, ${\varvec{\Lambda }}_\mathrm{C,Ind}$, appearing in the equation for ARE$_\mathrm{Ind}$, is obtained by eliminating the first column and row of $\bar{{\varvec{\Lambda }}}_\mathrm{C,Ind}$. The other asymptotic covariance matrices ${\varvec{\Lambda }}_\mathrm{P,Ind}$, ${\varvec{\Lambda }}_\mathrm{C,Dep}$ and ${\varvec{\Lambda }}_\mathrm{P,Dep}$ are obtained in the same manner.

Because the feature-independent labeling mechanism indicates that the labeling probability $\mathrm{P}[R=1|{\varvec{x}};{\varvec{\phi }}]=\mathrm{P}[R=1;{\varvec{\phi }}]=\gamma$ is the same for all ${\varvec{x}}$, the expectation above is equivalent to the standard argument of the Fisher information up to the factor $\gamma$ (e.g., see Magnus and Neudecker 1999). Therefore, the asymptotic variance is analytically represented as

$$\begin{aligned} \bar{{\varvec{\Lambda }}}_\mathrm{C,Ind} = \mathbf{L}_\mathrm{Ind}^{-1} , \end{aligned}$$

where $\mathbf{L}_\mathrm{Ind}$ is the matrix defined as

$$\begin{aligned} \frac{1}{\gamma } \begin{pmatrix} (\pi _1\pi _0)^{-1} &{} \mathbf {0}^T &{} \mathbf {0}^T &{} \mathbf {0}^T &{} \mathbf {0}^T\\ \mathbf {0} &{} \pi _1{\varvec{\Sigma }}_1^{-1} &{} O &{} O &{} O\\ \mathbf {0} &{} O &{} \displaystyle \frac{\pi _1}{2}{} \mathbf{D}'({\varvec{\Sigma }}_1\otimes {\varvec{\Sigma }}_1)^{-1}{} \mathbf{D} &{} O &{} O\\ \mathbf {0} &{} O &{} O &{} \pi _0{\varvec{\Sigma }}_0^{-1} &{} O \\ \mathbf {0} &{} O &{} O &{} O &{} \displaystyle \frac{\pi _0}{2}{} \mathbf{D}'({\varvec{\Sigma }}_0\otimes {\varvec{\Sigma }}_0)^{-1}{} \mathbf{D} \end{pmatrix} . \end{aligned}$$

Clearly $\bar{{\varvec{\Lambda }}}_\mathrm{C,Ind}$ is non-singular unless either ${\varvec{\Sigma }}_1$ or ${\varvec{\Sigma }}_0$ is singular.

Appendix B: Asymptotic covariance matrix of $\hat{{\varvec{\vartheta }}}_\mathrm{P,Ind}$

We derive the asymptotic covariance matrix of estimator $\hat{{\varvec{\vartheta }}}_\mathrm{P,Ind}$ under the feature-independent labeling mechanism. Together with the result in Appendix A, the asymptotic covariance matrix of the above estimator, denoted by $\bar{{\varvec{\Lambda }}}_\mathrm{P,Ind}$, is obtained by taking the expectation of the second derivative of the score function $S_\mathrm{P,Ind}({\varvec{\theta }})$ and taking its inverse. Because the information corresponding to the first term of $S_\mathrm{P,Ind}({\varvec{\theta }})$ has already been given by $\mathbf{L}_\mathrm{Ind}$, it suffices to calculate the information of the unlabeled data. That is,

$$\begin{aligned} \bar{{\varvec{\Lambda }}}_\mathrm{P,Ind} = (\mathbf{L}_\mathrm{Ind}+\mathbf{U}_\mathrm{Ind})^{-1} \end{aligned}$$

with $\mathbf{U}_\mathrm{Ind} = (1-\gamma )\int _{\mathbf {R}^d} {{\varvec{a}}}({\varvec{x}}){{\varvec{a}}}({\varvec{x}})'f({\varvec{x}};{\varvec{\theta }})d{\varvec{x}}$, where

$$\begin{aligned}&{{\varvec{a}}}({\varvec{x}}) = \begin{pmatrix} a_{\bullet }({\varvec{x}}) \\ a_1({\varvec{x}}){\varvec{c}}_1({\varvec{x}}) \\ a_0({\varvec{x}}){\varvec{c}}_0({\varvec{x}}) \end{pmatrix},\\&a_\ell ({\varvec{x}})=\frac{\pi _\ell f_\ell ({\varvec{x}};{\varvec{\theta }})}{\pi _1f_1({\varvec{x}};{\varvec{\theta }})+\pi _0f_0({\varvec{x}};{\varvec{\theta }})},\ \ {\varvec{c}}_\ell ({\varvec{x}}) = \begin{pmatrix} {\varvec{d}}_\ell ({\varvec{x}}) \\ -\frac{1}{2}D'{\varvec{v}}_\ell ({\varvec{x}}) \end{pmatrix} ,\\&{\varvec{d}}_\ell ({\varvec{x}})={\varvec{\Sigma }}_{\ell }^{-1}({\varvec{x}}-{\varvec{\mu }}_\ell ),\ \ {\varvec{v}}_\ell ({\varvec{x}})=\mathrm{vec}({\varvec{\Sigma }}_{\ell }^{-1}-{\varvec{d}}_\ell ({\varvec{x}}){\varvec{d}}_\ell ({\varvec{x}})'),\ \ \ell =1,0\\&\mathrm{and}\ \ a_{\bullet }({\varvec{x}}) = \frac{a_1({\varvec{x}})}{\pi _1}-\frac{a_0({\varvec{x}})}{\pi _0}. \end{aligned}$$

The form of $\mathbf{U}_\mathrm{Ind}$ is equivalent to a result given in Boldea and Magnus (2009) up to constant $(1-\gamma )$. Because integration in $\mathbf{U}_\mathrm{Ind}$ cannot be expressed in an analytic form, the Monte Carlo approximation is needed for its calculation.

Appendix C: Asymptotic covariance matrix of $\hat{{\varvec{\vartheta }}}_\mathrm{C,Dep}$

As well as the feature-independent labeling mechanism, we derive the asymptotic covariance matrix of the estimator of ${\varvec{\theta }}$ based on $S_\mathrm{C,Dep}({\varvec{\theta }})$ under the feature-dependent labeling mechanism. This situation differs from that for the feature-independent labeling mechanism in that term $1/\mathrm{P}[R=1|{\varvec{x}};{\varvec{\phi }}]$ is included in the score functions. As the labeling probability $\mathrm{P}[R=1|{\varvec{x}};{\varvec{\phi }}]$ is not constant for the feature-dependent labeling mechanisms, the information matrices do not have an explicit representation, unlike $\mathbf{L}_\mathrm{Ind}$. Therefore, a computation method such as the Monte Carlo integral is needed to calculate these information matrices.

The asymptotic variance of estimator $\hat{{\varvec{\vartheta }}}_\mathrm{C,Dep}$ is obtained as follows:

$$\begin{aligned} \bar{{\varvec{\Lambda }}}_\mathrm{C,Dep} = \mathbf{L}_\mathrm{Dep}^{-1}, \end{aligned}$$

where $\mathbf{L}_\mathrm{Dep}= \int _{\mathbf {R}^d}\left\{ \pi _1f_1({\varvec{x}};{\varvec{\theta }})\mathbf{B}_{1}({\varvec{x}})+\pi _0f_0({\varvec{x}};{\varvec{\theta }})\mathbf{B}_{0}({\varvec{x}}) \right\} d{\varvec{x}}$,

$\mathbf{B}_{\ell }({\varvec{x}}) = \displaystyle \frac{{{\varvec{b}}_\ell ({\varvec{x}}){\varvec{b}}_\ell ({\varvec{x}})'}}{\mathrm{P}[R=1|{\varvec{x}};{\varvec{\phi }}]}$, $\ell =0,1$, ${\varvec{b}}_{1}({\varvec{x}}) = \begin{pmatrix} \pi _1^{-1}\\ {\varvec{d}}_1({\varvec{x}}) \\ -\frac{1}{2}{\varvec{v}}_1({\varvec{x}})'{} \mathbf{D} \\ {\varvec{0}}_{d} \\ {\varvec{0}}_{d(d+1)/2} \end{pmatrix}$, ${\varvec{b}}_{0}({\varvec{x}}) = \begin{pmatrix} -\pi _0^{-1}\\ {\varvec{0}}_{d} \\ {\varvec{0}}_{d(d+1)/2} \\ {\varvec{d}}_0({\varvec{x}}) \\ -\frac{1}{2}{\varvec{v}}_0({\varvec{x}})'{} \mathbf{D} \end{pmatrix}$ and ${\varvec{0}}_d$ is the d-dimensional zero vector.

Appendix D: Asymptotic covariance matrix of $\hat{{\varvec{\vartheta }}}_\mathrm{P,Dep}$

Besides the direct calculation of the asymptotic covariance matrix of estimator $\hat{{\varvec{\vartheta }}}_\mathrm{P,Dep}$, it suffices to calculate the expectation of the second derivative of the second term on the right-hand side of $S_\mathrm{P,Dep}({\varvec{\theta }})$. Then, the asymptotic covariance matrix, denoted by $\bar{{\varvec{\Lambda }}}_\mathrm{P,Dep}$, is obtained from

$$\begin{aligned} \bar{{\varvec{\Lambda }}}_\mathrm{P,Dep} = (\mathbf{L}_\mathrm{Dep}+\mathbf{U}_\mathrm{Dep})^{-1}, \end{aligned}$$

where $\mathbf{U}_\mathrm{Dep} = \int _{\mathbf {R}^d} \frac{{{\varvec{a}}}({\varvec{x}}){{\varvec{a}}}({\varvec{x}})'}{\mathrm{P}[R=0|{\varvec{x}};{\varvec{\phi }}]}f({\varvec{x}};{\varvec{\theta }})d{\varvec{x}}.$