A feature selection method using improved regularized linear discriminant analysis

Abstract

Investigation of genes, using data analysis and computer-based methods, has gained widespread attention in solving human cancer classification problem. DNA microarray gene expression datasets are readily utilized for this purpose. In this paper, we propose a feature selection method using improved regularized linear discriminant analysis technique to select important genes, crucial for human cancer classification problem. The experiment is conducted on several DNA microarray gene expression datasets and promising results are obtained when compared with several other existing feature selection methods.

Appendices

Appendix A

In this section, we use cross-validation procedure to compute average classification accuracy using four distinct classifiers and the proposed feature selection method. Three datasets have been used for this purpose are SRBCT, MLL and Acute Leukemia. The classification accuracy using fold $k=5$ and fold $k=10$ are given in Tables 10, 11 and 12. It can be observed that the classification accuracy obtained by $k$-fold cross-validation procedure is comparably similar to the classification accuracy obtained in Tables 2-4.

Table 10 $k$-fold cross-validation using improved RLDA and four distinct classifiers on the SRBCT dataset

Table 11 $k$-fold cross-validation using improved RLDA and four distinct classifiers on the MLL dataset

Table 12 $k$-fold cross-validation using improved RLDA and four distinct classifiers on the Acute Leukemia dataset

Appendix B

In this appendix, we compare different values of regularization parameter with the proposed improved RLDA technique. In order to show this, we computed classification accuracy on four different values of $\alpha$ for RLDA technique. These are $\delta =[0.001,0.01,0.1,1]$, where $\alpha =\delta *\lambda _{\mathrm{W}}$ and $\lambda _{\mathrm{W}}$ is the maximum eigenvalue of within-class scatter matrix. We applied threefold cross-validation procedure on a number of datasets and shown the results in columns 2–5 of Table 11. The last column of the table denotes the classification accuracy using improved RLDA technique (Table 13).

Table 13 Classification accuracy (in percentage) of RLDA and improved RLDA

It can be observed from the table that the different values of the regularization parameter give different classification accuracies and therefore, the choice of the regularization parameter affects the classification performance. Thus, it is important to select the regularization parameter correctly to get the good classification performance. It can be observed that for all the datasets, the proposed technique is exhibiting promising results.

Appendix C

Corollary 1

The value of regularization parameter is non-negative; i.e., $\alpha \ge 0$ for $r_w \le r_t$, where $r_t =\mathrm{rank}({\mathbf{S}}_{\mathrm{T}} )$ and $r_w =\mathrm{rank}({\mathbf{S}}_{\mathrm{W}} )$.

Proof

From Eq. 2, we can write

$$\begin{aligned} J=\frac{{\mathbf{w}}^\mathrm{T}{\mathbf{S}}_{\mathrm{B}} {\mathbf{w}}}{{\mathbf{w}}^\mathrm{T}({\mathbf{S}}_{\mathrm{W}} +\alpha {\mathbf{I}}){\mathbf{w}}}, \end{aligned}$$

(11)

where ${\mathbf{S}}_{\mathrm{B}} \in {\mathbb {R}}^{r_t \times r_t }$ and ${\mathbf{S}}_{\mathrm{W}} \in {\mathbb {R}}^{r_t \times r_t }$. We can rearrange the above expression as

$$\begin{aligned} {\mathbf{w}}^\mathrm{T}{\mathbf{S}}_\mathrm{B} {\mathbf{w}}=J{\mathbf{w}}^\mathrm{T}({\mathbf{S}}_{\mathrm{W}} +\alpha {\mathbf{I}}){\mathbf{w}} \end{aligned}$$

(12)

The eigenvalue decomposition (EVD) of ${\mathbf{S}}_{\mathrm{W}}$ matrix (assuming $r_w <r_t )$ can be given as ${\mathbf{S}}_{\mathrm{W}} ={\mathbf{U\Lambda }}^2{\mathbf{U}}^\mathrm{T},$ where ${\mathbf{U}}\in {\mathbb {R}}^{r_t \times r_t }$ is an orthogonal matrix, ${{\varvec{\Lambda }}}^2=\small \left[ {{\begin{array}{l@{\quad }l} {{{\varvec{\Lambda }}}_w^2 } &{} 0 \\ 0 &{} 0 \\ \end{array} }} \right] \in {\mathbb {R}}^{r_t \times r_t }$ and ${{\varvec{\Lambda }}}_w =\mathrm{diag}(q_1^2 ,q_2^2 ,\ldots ,q_{r_w }^2 )\in {\mathbb {R}}^{r_w \times r_w }$ are diagonal matrices (as $r_w <r_t )$. The eigenvalues $q_k^2 >0$ for $k=1,2,\ldots ,r_w$. Therefore,

$$\begin{aligned}&{\mathbf{S}}_{\mathrm{W}}^{\prime } =( {{\mathbf{S}}_{\mathrm{W}} +\alpha {\mathbf{I}}})={\mathbf{UDU}}^\mathrm{T}, \hbox { where } {\mathbf{D}}={{\varvec{\Lambda }}}^2+\alpha {\mathbf{I}}\nonumber \\&\hbox {or}\,{\mathbf{D}}^{-1/2}{\mathbf{U}}^\mathrm{T}{\mathbf{S}}_{\mathrm{W}}^{\prime } {\mathbf{UD}}^{-1/2}={\mathbf{I}} \end{aligned}$$

(13)

The between class scatter matrix ${\mathbf{S}}_{\mathrm{B}}$ can be transformed by multiplying ${\mathbf{UD}}^{-1/2}$ on the right side and ${\mathbf{D}}^{-1/2}{\mathbf{U}}^\mathrm{T}$ on the left side of ${\mathbf{S}}_{\mathrm{B}}$ as ${\mathbf{D}}^{-1/2}{\mathbf{U}}^\mathrm{T}{\mathbf{S}}_{\mathrm{B}} {\mathbf{UD}}^{-1/2}$. The EVD of this matrix will give

$$\begin{aligned} {\mathbf{D}}^{-1/2}{\mathbf{U}}^\mathrm{T}{\mathbf{S}}_{\mathrm{B}} {\mathbf{UD}}^{-1/2}={\mathbf{ED}}_{\mathrm{B}} {\mathbf{E}}^{\mathbf{T}}, \end{aligned}$$

(14)

where ${\mathbf{E}}\in {\mathbb {R}}^{r_t \times r_t }$ is an orthogonal matrix and ${\mathbf{D}}_{\mathrm{B}} \in {\mathbb {R}}^{r_t \times r_t }$ is a diagonal matrix. Equation 14 can be rearranged as

$$\begin{aligned} {\mathbf{E}}^{\mathbf{T}}{\mathbf{D}}^{-1/2}{\mathbf{U}}^\mathrm{T}{\mathbf{S}}_{\mathrm{B}} {\mathbf{UD}}^{-1/2}{\mathbf{E}}={\mathbf{D}}_{\mathrm{B}}, \end{aligned}$$

(15)

Let the leading eigenvalue of ${\mathbf{D}}_{\mathrm{B}}$ is $\gamma$ and its corresponding eigenvector is ${\mathbf{e}}\in {\mathbf{E}}$. Then Eq. 15 can be rewritten as

$$\begin{aligned} {\mathbf{e}}^{\mathbf{T}}{\mathbf{D}}^{-1/2}{\mathbf{U}}^\mathrm{T}{\mathbf{S}}_{\mathrm{B}} {\mathbf{UD}}^{-1/2}{\mathbf{e}}={\gamma ,} \end{aligned}$$

(16)

The eigenvector ${\mathbf{e}}$ can be multiplied right side and ${\mathbf{e}}^\mathrm{T}$ on left side of Eq. 13, we get

$$\begin{aligned} {\mathbf{e}}^\mathrm{T}{\mathbf{D}}^{-1/2}{\mathbf{U}}^\mathrm{T}{\mathbf{S}}_{\mathrm{W}}^{\prime } {\mathbf{UD}}^{-1/2}{\mathbf{e}}=1 \end{aligned}$$

(17)

It can be seen from Eqs. 13 and 15 that matrix ${\mathbf{W}}={\mathbf{UD}}^{-1/2}{\mathbf{E}}$ diagonalizes both ${\mathbf{S}}_{\mathrm{B}}$ and ${\mathbf{S}}_{\mathrm{W}}^{\prime }$, simultaneously. Also vector ${\mathbf{w}}={\mathbf{UD}}^{-1/2}{\mathbf{e}}$ simultaneously gives $\gamma$ and unity eigenvalue in Eqs. 16 and 17. Therefore, ${\mathbf{w}}$ is a solution of Eq. 12. Substituting ${\mathbf{w}}={\mathbf{UD}}^{-1/2}{\mathbf{e}}$ in Eq. 12, we get $J=\gamma ;$ i.e., ${\mathbf{w}}$ is a solution of Eq. 12.

From Lemma 1, the maximum eigenvalue of expression $({\mathbf{S}}_{\mathrm{W}} +\alpha {\mathbf{I}})^{-1}{\mathbf{S}}_{\mathrm{B}} {\mathbf{w}}={\gamma }{\mathbf{w}}$ is $\gamma _m =\lambda _{\mathrm{max}} >0$ (i.e., real, positive and finite). Therefore, the eigenvectors corresponding to this positive $\gamma _m$ should also be in real hyperplane (i.e., the components of the vector ${\mathbf{w}}$ have to have real values). Since ${\mathbf{w}}={\mathbf{UD}}^{-1/2}{\mathbf{e}}$ with ${\mathbf{w}}$ to be in real hyperplane, we must have ${\mathbf{D}}^{-1/2}$ to be real.

Since ${\mathbf{D}}={{\varvec{\Lambda }}}^2+\alpha {\mathbf{I}}=\mathrm{diag}(q_1^2 +\alpha ,q_2^2 +\alpha ,\ldots ,q_{r_w }^2 +\alpha ,\alpha ,\ldots ,\alpha )$, we have

$$\begin{aligned} {\mathbf{D}}^{-1/2}&= \mathrm{diag}(1/\sqrt{q_1^2 +\alpha } ,1/\sqrt{q_2^2 +\alpha } ,\ldots ,1/\sqrt{q_{r_w }^2 +\alpha },\nonumber \\&\quad 1/\sqrt{\alpha },\ldots ,1/\sqrt{\alpha }). \end{aligned}$$

Therefore, the elements of ${\mathbf{D}}^{-1/2}$, must satisfy $1/\sqrt{q_k^2 +\alpha } >0$ and $1/\sqrt{\alpha }>0$ for $k=1,2,\ldots ,r_w$ (note $r_w <r_t )$; i.e., $\alpha$ cannot be negative or $\alpha >0$. Furthermore, if $r_w =r_t$ then matrix ${\mathbf{S}}_{\mathrm{W}}$ will be a non-singular matrix and its inverse will exist. In this case, regularization is not required and therefore $\alpha \!=\!0$. Thus, $\alpha \!\ge \! 0$ for $r_w \!\le \! r_t$. This concludes the proof. $\quad \square$