Isotonic boosting classification rules

Abstract

In many real classification problems a monotone relation between some predictors and the classes may be assumed when higher (or lower) values of those predictors are related to higher levels of the response. In this paper, we propose new boosting algorithms, based on LogitBoost, that incorporate this isotonicity information, yielding more accurate and easily interpretable rules. These algorithms are based on theoretical developments that consider isotonic regression. We show the good performance of these procedures not only on simulations, but also on real data sets coming from two very different contexts, namely cancer diagnostic and failure of induction motors.

Appendix

1.1 A.1 Theoretical justification for algorithms under the adjacent categories model

Let $\mathbf{x} \in \mathbb {R}^d,y\in \{1,\dots ,K\}, y_k^*=I_{\left[ y=k\right] },k=1,\dotsc ,K$ and assume the adjacent probabilities model (2). Denote further $F_1(\mathbf{x} )=0$, so the a posteriori probabilities are:

$$\begin{aligned} p_k(\mathbf{x} )=\frac{\exp \left( \sum _{j=1}^k F_j(\mathbf{x} )\right) }{\sum _{k=1}^K \exp \left( \sum _{j=1}^k F_j(\mathbf{x} )\right) },k=1,\dotsc ,K. \end{aligned}$$

Now, the expected log-likelihood is:

$$\begin{aligned} E l(F_2,\dots ,F_K)= E\left[ \sum _{k=2}^K y_k^* \left( \sum _{j=2}^k F_j(\mathbf{x} )\right) - \log \left( 1+\sum _{k=2}^K \exp \left( \sum _{j=2}^k F_j(\mathbf{x} )\right) \right) \right] . \end{aligned}$$

Conditioning on $\mathbf{x}$, the score vector $\mathbf{S}(x) =(s_k(\mathbf{x} ))$ for the population Newton algorithm is:

$$\begin{aligned} s_k(\mathbf{x} )= \frac{\partial E l(F_2(\mathbf{x} ),\dots ,F_K(\mathbf{x} ))}{\partial F_k(\mathbf{x} )}=E\left( \left. \sum _{j=k}^K (y_j^*-p_j(\mathbf{x} ))\right| \mathbf{x} \right) ,k=2,\dotsc ,K. \end{aligned}$$

The Hessian is a $(K-1)\times (K-1)$ matrix, $\mathbf{H} (\mathbf{x} )=(H_{km}(\mathbf{x} ))$, $2\le k,m\le K$, where each element $H_{km}(\mathbf{x} )$ is:

$$\begin{aligned} H_{km}(\mathbf{x} ) = \frac{\partial ^2 E l(F_2(\mathbf{x} ),\dots ,F_K(\mathbf{x} ))}{\partial F_k(\mathbf{x} ) \partial F_m(\mathbf{x} )}= \left\{ \begin{array}{l l} -(\sum _{j=m}^K p_j(\mathbf{x} ))(1-\sum _{j=k}^K p_j(\mathbf{x} )), &{} m\ge k\\ -(\sum _{j=k}^K p_j(\mathbf{x} ))(1-\sum _{j=m}^K p_j(\mathbf{x} )), &{} m < k \end{array} \right. \end{aligned}$$

If $\mathbf{W} (\mathbf{x} )=-diag(\mathbf{H} (\mathbf{x} ))$, a quasi-Newton update for the ASILB algorithm is:

$$\begin{aligned} \begin{bmatrix} F_2(\mathbf{x} ) \\ \vdots \\ F_K(\mathbf{x} ) \end{bmatrix} \leftarrow \begin{bmatrix} F_2(\mathbf{x} ) \\ \vdots \\ F_K(\mathbf{x} ) \end{bmatrix} + E_W\left( \left. \mathbf{W} ^{-1}(\mathbf{x} ) \mathbf{s} (\mathbf{x} )\right| \mathbf{x} \right) . \end{aligned}$$

The full Newton update, which is implemented in the AMILB algorithm, is:

$$\begin{aligned} \begin{bmatrix} F_2(\mathbf{x} ) \\ \vdots \\ F_K(\mathbf{x} ) \end{bmatrix} \leftarrow E_H\left( \left. \begin{bmatrix} F_2(\mathbf{x} ) \\ \vdots \\ F_K(\mathbf{x} ) \end{bmatrix} - \mathbf{H} ^{-1}(\mathbf{x} ) \mathbf{s} (\mathbf{x} )\right| \mathbf{x} \right) \end{aligned}$$

1.2 A.2 Theoretical justification for algorithms under the cumulative probabilities model

In this case we have to update the function $F(\mathbf{x} )$ and the parameters $\alpha _k,k=2,\dotsc ,K$. We will perform a “two step” update, first on $F(\mathbf{x} )$ and then on the $\alpha$ parameters.

Let us denote $\gamma _k(\mathbf{x} )=\sum _{j=k}^K p_j(\mathbf{x} ),k=1,\dotsc ,K$, and assume the cumulative probabilities model (3). For this model

$$\begin{aligned} \gamma _k(\mathbf{x} ) = \frac{\exp (\alpha _k+F(\mathbf{x} ))}{1+\exp (\alpha _k+F(\mathbf{x} ))},k=2,\dotsc ,K, \end{aligned}$$

with $\gamma _1(\mathbf{x} )=1$ and $\gamma _{K+1}(\mathbf{x} )=0.$

First, we perform the $F(\mathbf{x} )$ update. Here, as in the previous model, we consider a single observation as this step is used for updating the weights and the values to be adjusted. The expected log-likelihood is:

$$\begin{aligned} E l(F)= E\left( \sum _{k=1}^K y_k^* \log (\gamma _k(\mathbf{x} ) - \gamma _{k+1}(\mathbf{x} ))\right) . \end{aligned}$$

Conditioning on $\mathbf{x}$, the first and second derivatives for the population Newton algorithm are:

$$\begin{aligned} \frac{\partial E l(F(\mathbf{x} ))}{\partial F(\mathbf{x} )}= E\left( \left. \sum _{k=1}^K y_k^* [1-\gamma _k(\mathbf{x} ) - \gamma _{k+1}(\mathbf{x} )]\right| \mathbf{x} \right) , \end{aligned}$$

and

$$\begin{aligned} w(\mathbf{x} )= -\frac{\partial ^2 E l(F(\mathbf{x} ))}{\partial F(\mathbf{x} )^2} = E\left( \left. \sum _{k=1}^K y_k^* [\gamma _k(\mathbf{x} )(1-\gamma _k(\mathbf{x} )) + \gamma _{k+1}(\mathbf{x} )(1-\gamma _{k+1}(\mathbf{x} ))]\right| \mathbf{x} \right) , \end{aligned}$$

so that the Newton update for $F(\mathbf{x} )$ is

$$\begin{aligned} F(\mathbf{x} ) \leftarrow E_H\left( \left. F(\mathbf{x} ) + \frac{1}{w(\mathbf{x} )}\frac{\partial E l(F(\mathbf{x} ))}{\partial F(\mathbf{x} )}\right| \mathbf{x} \right) . \end{aligned}$$

As for the parameters $\alpha _k,k=2,\dotsc ,K$, let us denote $\varvec{\alpha }=(\alpha _2,\dots ,\alpha _K)'$. Now, we use all the $\mathbf{x} _i$ observations as in this case we are going to perform a Newton step for updating the $\alpha$’s which do not depend on $\mathbf{x}$. Then, the log-likelihood is:

$$\begin{aligned} l(\varvec{\alpha })= \sum _{i=1}^n \sum _{k=1}^K y_{k,i}^* \log (\gamma _k(\mathbf{x} _i) - \gamma _{k+1}(\mathbf{x} _i)). \end{aligned}$$

The score for the Newton algorithm $\mathbf{S} =(s_2,\dots ,s_K)'$ is:

$$\begin{aligned} s_k= \frac{\partial l(\varvec{\alpha })}{\partial \alpha _k} = \sum _{i=1}^n \left( \frac{y_{k,i}^*}{p_k(\mathbf{x} _i)}- \frac{y_{k-1,i}^*}{p_{k-1}(\mathbf{x} _i)}\right) \gamma _k(\mathbf{x} _i)(1-\gamma _k(\mathbf{x} _i)),k=2,\dotsc ,K. \end{aligned}$$

(4)

And the Hessian is a tri-diagonal symmetric $(K-1)\times (K-1)$ matrix $\mathbf{H} =(H_{km})$ with $H_{km} = \frac{\partial ^2 l(\varvec{\alpha })}{\partial \alpha _k\partial \alpha _m},$ for $2\le k,m\le K$, such that

$$\begin{aligned} H_{kk}&= -\sum _{i=1}^n \gamma _k(\mathbf{x} _i)(1-\gamma _k(\mathbf{x} _i)) \left[ \frac{y_k^*}{p_k^2(\mathbf{x} _i)}(p_k^2(\mathbf{x} _i)+\gamma _{k+1}(\mathbf{x} _i)(1-\gamma _{k+1}(\mathbf{x} _i)))\right. \nonumber \\&\left. \quad +\frac{y_{k-1}^*}{p_{k-1}^2(\mathbf{x} _i)} (p_{k-1}^2(\mathbf{x} _i)+\gamma _{k-1}(\mathbf{x} _i)(1-\gamma _{k-1}(\mathbf{x} _i)))\right] \end{aligned}$$

(5)

$$\begin{aligned} H_{k,k-1}&=H_{k-1,k}= \sum _{i=1}^n \frac{y_{k-1,i}^*}{p_{k-1}^2(\mathbf{x} _i))}\gamma _k(\mathbf{x} _i)(1-\gamma _k(\mathbf{x} _i))\gamma _{k-1}(\mathbf{x} _i)(1-\gamma _{k-1}(\mathbf{x} _i)) \end{aligned}$$

(6)

$$\begin{aligned} H_{km}&=H_{mk}=0 \text { otherwise.} \end{aligned}$$

(7)

In these conditions the Newton update is $\varvec{\alpha } \leftarrow \varvec{\alpha } - \mathbf{H} ^{-1} \mathbf{S} .$

1.3 A.3 Full numerical results for the simulations performed

This subsection contains the Tables showing the full numerical mean results for TMP, MAE and AUC for the two sets of simulations. In Tables 7, 8 and 9 appear the results for the first set of simulations performed under model 2 for the different F functions appearing in Table 3. Tables 10, 11 and 12 contain the mean results for the simulations performed under the uniform order-restricted predictors scheme. In all cases the best results appear in bold. Notice that there are no results for CSILB and CMILB when $K=2$ as in that case those algorithms coincide with ASILB and AMILB. For this case the TMP and MAE values also coincide. They are given in both tables for completeness.

Table 7 Mean TMP for the first simulation scheme for different classification rules, number of groups K, predictors d and functions F

Table 8 Mean MAE for the first simulation scheme for different classification rules, number of groups K, predictors d and functions F

Table 9 Mean AUC for the first simulation scheme for different classification rules, number of groups K, predictors d and functions F

Table 10 Mean TMP for different classification rules, number of groups K, predictors d and training sample sizes n, for the second set of simulations

Table 11 Mean MAE for different classification rules, number of groups K, predictors d and training sample sizes n, for the second set of simulations

Table 12 Mean AUC for different classification rules, number of groups K, predictors d and training sample sizes n, for the second set of simulations