Classification tree algorithm for grouped variables

Abstract

We consider the problem of predicting a categorical variable based on groups of inputs. Some methods have already been proposed to elaborate classification rules based on groups of variables (e.g. group lasso for logistic regression). However, to our knowledge, no tree-based approach has been proposed to tackle this issue. Here, we propose the Tree Penalized Linear Discriminant Analysis algorithm (TPLDA), a new-tree based approach which constructs a classification rule based on groups of variables. It consists in splitting a node by repeatedly selecting a group and then applying a regularized linear discriminant analysis based on this group. This process is repeated until some stopping criterion is satisfied. A pruning strategy is proposed to select an optimal tree. Compared to the existing multivariate classification tree methods, the proposed method is computationally less demanding and the resulting trees are more easily interpretable. Furthermore, TPLDA automatically provides a measure of importance for each group of variables. This score allows to rank groups of variables with respect to their ability to predict the response and can also be used to perform group variable selection. The good performances of the proposed algorithm and its interest in terms of prediction accuracy, interpretation and group variable selection are loud and compared to alternative reference methods through simulations and applications on real datasets.

Appendices

Time complexity of TPLDA

In the following section, the maximal time complexity at a node t of the TPLDA method is detailed. We assume that there are:

• $n_{t}$ observations in the node t,

• J groups of variables denoted $X^j$, for $j=1,\ldots ,J$ and

• the group ${\mathbf {X}}^j$, with $j=1,\ldots ,J$, includes $d_j$ variables such as ${\mathbf {X}}^j = (X_{j_1},X_{j_2},\ldots ,X_{j_{d_j}})$.

To split a node t including $n_{t}$ observations, TPLDA uses the following two steps:

• Step 1: within group PLDA.

  For any group ${\mathbf {X}}^j$ of variables, with $j=1,\ldots ,J$, a PLDA is applied on the node t and the shrinkage parameter $\lambda _j$ is selected by cross-validation.

• Step 2: choosing the splitting group.

  For any group ${\mathbf {X}}^j$ of variables, with $j=1,\ldots ,J$, TPLDA computes the penalized decrease in node impurity resulting from splitting on group j and selects the group that maximizes it.

The time complexity of these steps are detailed below. Consider the group j, with $j=1,\ldots ,J$.

$\textit{Complexity when performing PLDA on group}\, j$:

PLDA computation steps are described in the original paper (Witten and Tibshirani 2011). We detailed here its maximal time complexity:

• Complexity for constructing the estimated between covariance matrix $\widehat{B}^j_{t}$ is $\mathcal {O}\left( n_{t}d_j^2\right)$.

• Complexity for constructing the diagonal positive estimate of the within covariance matrix ${\widehat{\Sigma }}^j_{t}$ is $\mathcal {O}\left( n_{t}d_j\right)$.

• Complexity of the eigen analysis of $({\widehat{\Sigma }}^j_{t})^{-1}\widehat{B}^j_{t}$ is $\mathcal {O}\left( d_j^2\right)$.

• Complexity of the eigen analysis of $(\widehat{B}^j_{t})^{-1}\widehat{B}^j_{t}$ and the research for the dominant eigenvector is $\mathcal {O}\left( d_j^2\right)$.

• Complexity for estimating the penalized discriminant vector ${\hat{\beta }}^j$ by performing M iterations of the minimization-maximization algorithm (Lange et al. 2000) is $\mathcal {O}\left( Md_j^2\right)$.

$\Rightarrow$ So the maximal time complexity of performing PLDA on the group j in the node t is $\mathcal {O}\left( n_{t}d_j^2\right) + \mathcal {O}\left( n_{t}d_j\right) + \mathcal {O}\left( d_j^2\right) + \mathcal {O}\left( Md_j^2\right) = \mathcal {O}\left( n_{t}d_j^2\right)$ by supposing that $M< n_{t}$.

$\textit{Complexity when selecting of the shrinkage parameter}\, \lambda _j$:

The value of shrinkage parameter $\lambda _j$ is determined by using a K-fold cross-validation and a grid $\lbrace v_1,\ldots ,v_L \rbrace$ containing L values for $\lambda _j$. The maximal time complexity of this step is detailed below:

• Complexity for dividing the $n_{t}$ observations in the node t into K disjoint samples $\lbrace S_1,\ldots ,S_K\rbrace$ is $\mathcal {O}\left( n_{t}\right)$.

• For each fold k, $k=1,\ldots ,K$ and each value $v_{\ell }$, $\ell =1,\ldots ,L$:

  • Complexity for performing a PLDA on $t\setminus S_k$ (i.e. all the disjoint sets $\lbrace S_1,\ldots ,S_K\rbrace$ excepted $S_k$) with $\lambda _j=v_{\ell }$ is $\mathcal {O}\left( \frac{K-1}{K}n_{t}d_j^2\right)$.

  • Complexity for predicting the class of each observation in $S_k$ using the resulted PLDA model computed on $t\setminus S_k$ is $\mathcal {O}\left( \frac{n_{t}}{K}d_j\right)$.

  • Complexity for computing the penalized decrease $\Delta _j(t, v_{\ell })$ in node impurity is $\mathcal {O}\left( n_{t}\right)$.

• Complexity for choosing the value in the grid $\lbrace v_1,\ldots ,v_L \rbrace$ which maximizes the penalized decrease in node impurity is $\mathcal {O}\left( 1\right)$.

$\Rightarrow$ So the complexity for selecting the value of $\lambda _j$ is $\mathcal {O}\left( n_{t}\right) + \mathcal {O}\left( L(K-1)n_{t}d_j^2\right) + \mathcal {O}\left( Ln_{t}d_j\right) + \mathcal {O}\left( Ln_{t}\right) + \mathcal {O}\left( 1\right) = \mathcal {O}\left( L(K-1)n_{t}d_j^2\right)$.

Complexity when choosing the splitting group

TPLDA selects among the J estimated splits the one which maximizes the impurity decrease. The complexity of this step is $\mathcal {O}\left( 1\right)$.

Consequently, the maximal time complexity of TPLDA at a node t is in the worst case $\mathcal {O}\left( Jn_{t}d_{\max }^2\right) + \mathcal {O}\left( JL(K-1)n_{t}d_{\max }^2\right) + \mathcal {O}\left( 1\right) = \mathcal {O}\left( JLKn_{t}d_{\max }^2\right)$ with $d_{\max }= \max _j(d_j)$.

Additional figures about the illustration of the TPLDA method on a simple example

Figure 13 displays the two trees built by CART and TPLDA in the simple example used to illustrate TPLDA in Sect. 2.4. As mentioned previously, in this example, the TPLDA tree is much easier than the CART tree. Moreover, the simple TPLDA tree is as accurate as the complex CART tree (TPLDA AUC $=$ 0.90, CART AUC $=$ 0.89).

Fig. 13

The two trees associated to the TPLDA and CART partitions displayed in Fig. 2. Circles define the nodes and the figure in each node indicates the node label. The splitting rule is denoted below each node

Additional information about the numerical experiments

This section provides additional results and figures about the simulation studies.

1.1 Justification for using PLDA instead of FDA in the splitting process

First of all, here we discuss the choice of using PLDA instead of FDA in the splitting process. In TPLDA, PLDA is replaced by FDA. This modified TPLDA is named TLDA and is applied on each sample of the first three experiments.

Table 9 displays the simulation results for TLDA in comparison with TPLDA. First, when data are not grouped, TPLDA and TLDA give almost the same results and can be then used interchangeably. Indeed, when the group size equals 1 and if the regularized parameter in the PLDA problem (3) is set to zero, the FDA problem and the PLDA problem are identical.

Table 9 Comparison between TLDA and TPLDA: simulation results

In the second and the third experiment, TLDA underperforms TPLDA. This can be explained by the fact that FDA performs badly in small nodes i.e. in the nodes where the number of observations is small relative to the size of some groups of variables (Shao et al. 2011; Friedman 1989; Xu et al. 2009; Bouveyron et al. 2007). Yet, tree elaboration is based on a recursive splitting procedure which creates nodes that becomes smaller and smaller whereas the sizes of input groups remain unchanged. Then FDA may not be appropriate for estimating recursively the hyperplane splits. This is well illustrated by the performances of TLDA in the third experiment where the groups of input variables are large compared to the number of observations in the training sample. Indeed, in the first split, the FDA used to split the entire data space overfits the training set. This can be seen in Fig. 14: the training misclassification error decreases much faster for TLDA and becomes smaller than the Bayes error from the first split while the test misclassification error for TLDA remains stable. Consequently, after applying the pruning procedure which removes the less informative nodes, the final TLDA tree is trivial in at least 25 % of the simulations (Table 9). Conversely, TPLDA does not seem to be affected by the high-dimension. Then, PLDA overcomes the weakness of FDA in high-dimensional situations.

Fig. 14

Misclassification error estimate according to the tree depth on the training set (left) and on the validation set (right) in experiment 3. The dotted lines denote the values of the Bayes error (Bayes error $=$ 10%)

1.2 Sensitivity to the pruning strategy for CART

Here, the performances of CART when using the cost-complexity pruning strategy are compared to those obtained by using the proposed pruning strategy based on the tree depth, in the first three experiments. The approach using the proposed pruning strategy is named CARTD (while the approach using the cost-complexity pruning is named CART). The results are given in Table 10. Figure 15 displays the group selection frequencies of CART and CARTD. Overall, the two pruning methods lead to similar CART trees and so similar classification rules. Indeed, the predictive performances and the tree depth are very close. CARTD trees may be slightly smaller. Moreover, the group selection frequencies do not really differ: they are lightly higher when using the proposed pruning strategy based on the depth. Thus, CART performances do not seem to be sensitive to the choice of one of the two pruning methods.

Table 10 Performances of CART and CARTD

Fig. 15

Group selection frequency (in %) for CART according to the pruning strategy in the first three experiments

1.3 Additional results

Table 11 displays the simulation results for TPLDA, TLDA, CART and GL for the fourth and fifth scenarios. As previously, TLDA underperforms TPLDA. GL and CART overperform slightly TPLDA when no penalty function is used.

Table 11 Additional simulation results

1.4 Additional figures

Additional figures about the simulation studies are displayed in this subsection. Figure 16 displays the predictive performances of TPLDA, CART and GL in the first three experiments. Figure 17 shows the distribution of the importance score for each group in the first three experiments.

Fig. 16

Predictive performances of the assessed methods: boxplots of the AUC for the first three experiments

Fig. 17

Distribution of the importance score for each group in the first three experiments

Table 12 Choice of the clustering parameters for each dataset

Additional information about the application to gene expression data

Following Lee and Batzoglou (2003), we assume that each independent component refers to a putative biological process and that a group of genes is then created for each independent component. For a given independent component, the most important genes are the genes with the largest loads in absolute terms. Then, the group of genes associated to the given independent component includes the $C\%$ of genes with the largest loads in absolute terms. The number of groups J (or equivalently the number of independent components) and the threshold parameter C are tuning parameters.

Table 13 The datasets after applying data preprocessing and clustering genes into 50 groups

For each dataset, several values for the clustering parameters (J,C) are chosen in order to assess the sensitivity of the predictive performances of the methods TPLDA, CART, GL and SCRDA to the values of these parameters. For each dataset we show the results for two couples (J,C) (Table 12). In each dataset, genes are clustered into 15 or 50 non-mutually exclusive groups (Tables 7 and 13). Table 14 shows the results when genes are clustered into 50 groups of equal size. These results are not significantly different from those obtained when using 15 groups (Table 8).

Table 14 Average error rate for 500 samples when genes are clustered into 50 groups. For each scenario, the two methods with the best predictive results regarding the AUC are written in bold