The sparse signomial classification and regression model

Abstract

Kernel-based methods (KBMs) such as support vector machines (SVMs) are popular data mining tools for solving classification and regression problems. Due to their high prediction accuracy, KBMs have been successfully used in various fields. However, KBMs have three major drawbacks. First, it is not easy to obtain an explicit description of the discrimination (or regression) function in the original input space and to make a variable selection decision in the input space. Second, depending on the magnitude and numeric range of the given data points, the resulting kernel matrices may be ill-conditioned, with the possibility that the learning algorithms will suffer from numerical instability. Although data scaling can generally be applied to deal with this problem and related issues, it may not always be effective. Third, the selection of an appropriate kernel type and its parameters can be a complex undertaking, with the choice greatly affecting the performance of the resulting functions. To overcome these drawbacks, we present here the sparse signomial classification and regression (SSCR) model. SSCR seeks a sparse signomial function by solving a linear program to minimize the weighted sum of the ℓ ₁-norm of the coefficient vector of the function and the ℓ ₁-norm of violation (or loss) caused by the function. SSCR employs the signomial function in the original variables and can therefore explore the nonlinearity in the data. SSCR is also less sensitive to numerical values or numeric ranges of the given data and gives a sparse explicit description of the resulting function in the original input space, which will be useful for the interpretation purpose in terms of which original input variables and/or interaction terms are more meaningful than others. We also present column generation techniques to select important signomial terms in the classification and regression processes and explore a number of theoretical properties of the proposed formulation. Computational studies demonstrate that SSCR is at the very least competitive and can even perform better compared to other widely used learning methods for classification and regression.

Appendix

Proof of Theorem 1

The result can be established by showing that the well-known maximum independent set problem, which is known to be $\mathcal{NP}$-hard (Nemhauser and Wolsey 1988), is a special case of CGP. For a given undirected simple graph G=(V,E), and an independent set of G is a subset U of V such that (i,j)∉E for every pair of nodes i,j∈U. The problem is to find a maximum cardinality independent set of G.

For each node v∈V, we define an n-dimensional positive vector x _v with components x _vk for all k∈V, such that x _vk =1 for k≠v and x _vk =1/n ³, where n=|V|. We also define, for each edge e∈E, an n-dimensional vector y _e with components y _ek for all k∈V, such that y _ek =1/n ⁶ if k=i or k=j and y _ek =1 otherwise, where e=(i,j). Let S ₁={x _v |v∈V} and let S ₂={y _e |e∈E}. We specify the four parameters of D as d _min =0, d _max =1, T=1, and L=n. The set D is then the set of all n-dimensional binary vectors. We now set a ₁(x _v ) for each v∈V to be equal to 1, and set each element of a ₂(y _e ) for each e∈E to be n and define a special case of CGP with S ₁ and S ₂ along with a ₁(x _v ) for each v∈V, a ₂(y _e ) for each e∈E, and the four parameters of D. This special case of CGP is the problem of maximizing the function $z(\mathbf{d}) = \sum_{i \in V} (1/n^{3})^{d_{i}} - n(\sum_{(i,j) \in E} (1/n^{6})^{d_{i}} (1/n^{6})^{d_{j}})$ over all $\mathbf{d} \in\mathbb{B}^{n}$.

Now, we show that a maximum independent set U, such that |U|=K exists for some positive integer K<n if and only if $K \leq\hat{z} < K+1$, where $\hat{z} = \max\{z(\mathbf{d}) | \mathbf {d} \in\mathbb{B}^{n} \}$. Suppose that we have a maximum independent set U of G such that |U|=K. Then, for the vector d, such that d _i =0 if i∈U and d _j =1 otherwise z(d)≥K+1/n ³(n−K)−n|E|/n ⁶≥K and z(d)≤K+1/n ³(n−K)<K+1. Now, suppose that we have a binary vector d, such that $K \leq\hat{z} < K+1$ for some positive integer such that K<n. This means that at least one of d _i and d _j should be equal to 1 if and only if (i,j)∈E, otherwise, $\hat{z} \leq0$. Moreover, the number of zero elements of d is equal to K. If we define U={i∈V|d _i =0}, then U should be an independent set of cardinality K. Therefore, the result follows. □

Proof of Theorem 2

The result can also be established by showing that the maximum independent set problem is a special case of CGP with S ₁=S ₂. For any given instance of the maximum independent set problem, we define the vectors x _v , v∈V and y _e , e∈E as in the proof of Theorem 1. Let S ₁=S ₂={x _v ,v∈V}∪{y _e ,e∈E}. In addition to a ₁(x _v ) for each v∈V and a ₂(y _e ) for each e∈E defined in the proof of Theorem 1, we define a ₁(y _e )=0 for each e∈E and a ₂(x _v )=0 for each v∈V. We then can define a special case of CGP with S ₁, S ₂, a ₁, a ₂ along with the four parameters of D specified as those in the proof of Theorem 1. Observe that this special case of CGP with S ₁=S ₂ is the same problem as that discussed in the proof of Theorem 1. Therefore, the result follows. □