On support vector machines under a multiple-cost scenario

Abstract

Support vector machine (SVM) is a powerful tool in binary classification, known to attain excellent misclassification rates. On the other hand, many realworld classification problems, such as those found in medical diagnosis, churn or fraud prediction, involve misclassification costs which may be different in the different classes. However, it may be hard for the user to provide precise values for such misclassification costs, whereas it may be much easier to identify acceptable misclassification rates values. In this paper we propose a novel SVM model in which misclassification costs are considered by incorporating performance constraints in the problem formulation. Specifically, our aim is to seek the hyperplane with maximal margin yielding misclassification rates below given threshold values. Such maximal margin hyperplane is obtained by solving a quadratic convex problem with linear constraints and integer variables. The reported numerical experience shows that our model gives the user control on the misclassification rates in one class (possibly at the expense of an increase in misclassification rates for the other class) and is feasible in terms of running times.

Appendix A: Derivation of the CSVM

In this section, the detailed steps to build the CSVM formulation are shown. For that, suppose that we are given the mixed-integer quadratic model

$$\begin{aligned} \min _{\omega , \beta , \xi ,z}&\omega ^\top \omega + {C_+\sum \limits _{i \in I : y_i =+1} \xi _i + C_-\sum \limits _{i \in I : y_i =-1} \xi _i }&\\ \text {s.t.}&y_i(\omega ^\top x_i + \beta ) \ge 1 - \xi _i,&i \in I \\&\xi _i \ge 0&i \in I\\&y_j(\omega ^\top x_j + \beta ) \ge 1 - M_1(1-z_j),&j \in J \\&z_j \in \{ 0,1\}&j \in J\\&{\hat{p}_\ell \ge {p_{0}^*}_{\ell }}&\ell \in L. \end{aligned}$$

Hence, the problem above can be rewritten as

$$\begin{aligned} \begin{array}{lllllll} \min _{{z}} &{} &{} &{} {\min _{{\omega },\beta ,{\xi }} }&{} {{\omega }^\top {\omega } + {C_+\sum \limits _{i \in I : y_i =+1} \xi _i + C_-\sum \limits _{i \in I : y_i =-1} \xi _i } }\\ \text{ s.t. } &{} z_j \in \{0,1\} &{} j \in J &{} {\text{ s.t. }} &{} {y_i \left( {\omega }^\top {x}_i + \beta \right) \ge 1 - \xi _i} &{} {i \in I}\\ &{} {\hat{p}_\ell \ge {p_{0}^*}_{\ell }} &{} \ell \in L &{} &{}y_j\left( \omega ^\top x_j + \beta \right) \ge 1 - M_1(1-z_j),&{} j \in J \\ &{} &{} &{} &{} { \xi _i \ge 0} &{} {i \in I }. \end{array} \end{aligned}$$

We first develop the expression of the dual for the linear case and then we show how the kernel trick applies. As a previous step we should consider the variables z as fixed. Hence, having those variables fixed, the inner problem is rewritten as:

$$\begin{aligned} \begin{array}{llll} {\min _{{\omega },\beta ,{\xi }} }&{} {{\omega }^\top {\omega } + {C_+\sum \limits _{i \in I : y_i =+1} \xi _i + C_-\sum \limits _{i \in I : y_i =-1} \xi _i } }\\ {\text{ s.t. }} &{} {y_i \left( {\omega }^\top {x}_i + \beta \right) \ge 1 - \xi _i} &{} {i \in I}\\ &{}y_j\left( \omega ^\top x_j + \beta \right) \ge 1,&{} j \in J : z_j =1 \\ &{}y_j\left( \omega ^\top x_j + \beta \right) \ge 1 - M_1,&{} j \in J : z_j = 0\\ &{} { \xi _i \ge 0} &{} {i \in I }. \end{array} \end{aligned}$$

As $M_1$ is a large number, the fourth constraints always result feasible, so they can be removed. Also, we can denote $\{j \in J : z_j =1\}$ by J(z), obtaining

$$\begin{aligned} \begin{array}{llll} {\min _{{\omega },\beta ,{\xi }} }&{} {{\omega }^\top {\omega } + {C_+\sum \limits _{i \in I : y_i =+1} \xi _i + C_-\sum \limits _{i \in I : y_i =-1} \xi _i } }\\ {\text{ s.t. }} &{} {y_i \left( {\omega }^\top {x}_i + \beta \right) \ge 1 - \xi _i} &{} {i \in I}\\ &{}y_j\left( \omega ^\top x_j + \beta \right) \ge 1,&{} j \in J(z) \\ &{} { \xi _i \ge 0} &{} {i \in I }. \end{array} \end{aligned}$$

Hence, we can build the Lagrangian

$$\begin{aligned} \mathcal {L}(\omega ,\beta ,\xi )= & {} {{\omega }^\top {\omega } + {C_+\sum \limits _{i \in I : y_i =+1} \xi _i + C_-\sum \limits _{i \in I : y_i =-1} \xi _i }} \\&- \sum \limits _{s\in I} \lambda _s (y_s(\omega ^\top x_s+\beta ) -1 + \xi _s) \\&- \sum \limits _{t\in J(z)} \mu _t (y_t(\omega ^\top x_t+\beta ) -1) - \sum _{i' \in I} \delta _{i'} \xi _{i'} \end{aligned}$$

The KKT conditions are, therefore

$$\begin{aligned} \begin{array}{llllll} \dfrac{\partial \mathcal {L}}{\partial \omega } = 0 &{} \Rightarrow &{} {\omega } &{} = &{} \sum \limits _{s \in I} (\lambda _s/2) y_s {x}_s+ \sum \limits _{t \in J(z)} (\mu _t/2) y_t {x}_t \\ \dfrac{\partial \mathcal {L}}{\partial \beta } = 0 &{} \Rightarrow &{} 0 &{} = &{} \sum \limits _{s \in I} \lambda _s y_s + \sum \limits _{t \in J(z)} \mu _t y_t \\ \dfrac{\partial \mathcal {L}}{\partial \xi _i} = 0 &{} \Rightarrow &{} 0 &{} = &{} -\lambda _i -\delta _i + C_+ &{} i \in I:y_i =+1\\ \dfrac{\partial \mathcal {L}}{\partial \xi _i} = 0 &{} \Rightarrow &{} 0 &{} = &{} -\lambda _i -\delta _i + C_- &{} i \in I:y_i =-1\\ &{} &{} 0 &{} \le &{} \lambda _{i} &{} i \in I\\ &{} &{} 0 &{} \le &{} \mu _t &{} t \in J(z)\\ &{} &{} 0 &{} \le &{} \delta _{i} &{} i \in I \end{array} \end{aligned}$$

Note that we can replace, without loss of generality, $\lambda _s/2$, $\mu _t/2$ by $\lambda _s$ and $\mu _t$, respectively. Then, in the condition ${\partial \mathcal {L}}/{\partial \beta } = 0$ we have

$$\begin{aligned} 0 = \sum \limits _{s \in I} 2\lambda _s y_s + \sum \limits _{t \in J(z)} 2\mu _t y_t, \end{aligned}$$

that can be simplified to

$$\begin{aligned} 0 = \sum \limits _{s \in I} \lambda _s y_s + \sum \limits _{t \in J(z)} \mu _t y_t, \end{aligned}$$

as stated. In addition, the condition ${\partial \mathcal {L}}/{\partial \xi _i} = 0$ is transformed into

$$\begin{aligned} 0 = -2\lambda _i - \delta _i + C_+, \quad i \in I:y_i=+1 \end{aligned}$$

and

$$\begin{aligned} 0 = -2\lambda _i - \delta _i + C_-, \quad i \in I:y_i=-1. \end{aligned}$$

Furthermore, since these results must be equivalent to the case if we had maintained the previously removed constraint, we have $\mu _t = 0$ when $z_t=0, \quad t \in J$ and $\mu _t \ge 0$ when $z_t=1, \quad t \in J$. This can be summarized as $0 \le \mu _t \le M_2z_t,\quad t \in J$. Also, as usual, $\delta _i$ is removed since we add

$$\begin{aligned} 0 \le \lambda _i \le C_+/2, \quad i \in I:y_i=+1 \end{aligned}$$

and

$$\begin{aligned} 0 \le \lambda _i \le C_-/2, \quad i \in I:y_i=-1, \end{aligned}$$

as we know that $\delta _i \ge 0$. Therefore, the KKT conditions result:

$$\begin{aligned} \begin{array}{llll} {\omega } &{} = &{} \sum \limits _{s \in I} \lambda _s y_s {x}_s+ \sum \limits _{t \in J} \mu _t y_t {x}_t \\ 0 &{} = &{} \sum \limits _{s \in I} \lambda _s y_s + \sum \limits _{t \in J} \mu _t y_t \\ 0 &{} \le &{} \lambda _s \le C_+/2 &{} s\in I:y_i=+1 \\ 0 &{} \le &{} \lambda _s \le C_-/2 &{} s\in I:y_i=-1 \\ 0 &{} \le &{} \mu _t \le {M_2} z_t&{} t \in J. \end{array} \end{aligned}$$

Note that we have replaced all the J(z) by J using the previous clarification.

Thus, substituting the previous expressions into the second optimization problem, the partial dual of such problem can be calculated, yielding

$$\begin{aligned} \begin{array}{lllll} \min \limits _{{z}} &{} &{} &{} {\min \limits _{{\lambda },{\mu }, \beta , {\xi }} } &{} \left( \sum \limits _{s \in I} \lambda _s y_s {x}_s+ \sum \limits _{t \in J} \mu _t y_t {x}_t \right) ^\top \left( \sum \limits _{s \in I} \lambda _s y_s {x}_s+ \sum \limits _{t \in J} \mu _t y_t {x}_t \right) \\ &{} &{} &{} &{} {+ \, \, C_+\sum \limits _{i \in I : y_i =+1} \xi _i + C_-\sum \limits _{i \in I : y_i =-1} \xi _i } \\ \text{ s.t. } &{} z_j \in \{0,1\} &{} j \in J &{} {\text{ s.t. }} &{} {y_i \left( \left( \sum \limits _{s \in I} \lambda _s y_s {x}_s+ \sum \limits _{t \in J} \mu _t y_t {x}_t\right) ^\top {x}_i + \beta \right) }\\ &{}&{}&{}&{}{\ge 1 - \xi _i} \quad {i \in I}\\ &{} {\hat{p}_\ell \ge {p_{0}^*}_{\ell }} &{} \ell \in L &{} &{} { y_j\left( \left( \sum \limits _{s \in I} \lambda _s y_s {x}_s+ \sum \limits _{t \in J} \mu _t y_t {x}_t\right) ^\top {x}_j + \beta \right) }\\ &{}&{}&{}&{} {\ge 1 -{M_1}(1-z_j)} \quad {j \in J} \\ &{} &{} &{} &{} { \xi _i \ge 0} \quad {i \in I }\\ &{} &{} &{} &{} {\sum \limits _{i \in I} \lambda _i y_i + \sum \limits _{j \in J} \mu _j y_j = 0}\\ &{} &{} &{} &{} {{ 0 \le \lambda _i \le C_+/2} \quad {i \in I:y_i=+1 }}\\ &{} &{} &{} &{} {{ 0 \le \lambda _i \le C_-/2} \quad {i \in I:y_i=-1 }}\\ &{} &{} &{} &{} { 0 \le \mu _j \le {M_2}z_j} \quad {j \in J}. \end{array} \end{aligned}$$

Finally, since this problem only depends on the observation via the inner product, we can use the kernel trick and Problem (CSVM) is obtained.