Fast support vector clustering

Cluster analysis is a fundamental problem in pattern recognition where objects are categorized into groups or clusters based on pairwise similarities between those objects such that two criteria, homogeneity and separation, are achieved [21]. Two challenges in the task of cluster analysis are (1) dealing with complicated data with nested or hierarchy structures inside; and (2) automatically detecting the number of clusters. Recently, support-based clustering, e.g., support vector clustering (SVC) [1], has drawn a significant research concern because of its applications in solving the difficult and diverse clustering or outlier detection problem [1, 2, 8, 10, 11, 15, 23]. These clustering methods have two main advantages comparing with other clustering methods: (1) ability to generate the clustering boundaries with arbitrary shapes and automatically discover the number of clusters; and (2) capability to handle well the outliers.

Support-based clustering methods always undergo two phases. In the first phase, the domain of novelty, e.g., optimal hypersphere [1, 9, 22] or hyperplane [18], is discovered in the feature space. The domain of novelty when mapped back into the input space will become a set of contours tightly enclosing data which can be interpreted as cluster boundaries. However, this set of contours does not specify how to assign a data sample to its cluster. In addition, the computational complexity of the current solvers [3, 7] to find out the domain of novelty is often over-quadratic [4]. Such a computational complexity impedes the usage of support-based clustering methods for the real-world datasets. In the second phase, namely clustering assignment, based on the geometry information carried in the resultant set of contours harvested from the first phase, data samples are appointed to their clusters. Several works have been proposed for improving cluster assignment procedure [2, 8, 11, 15, 23].

Recently, stochastic gradient descent (SGD) frameworks [6, 19, 20] have emerged as building blocks to develop the learning methods for efficiently handling the large-scale dataset. SGD-based algorithm has the following advantages: (1) very fast; (2) ability to run in online mode; and (3) not requiring to load the entire dataset to the main memory in training. In this paper, we conjoin the advantages of SGD with support-based clustering. In particular, we propose to use the optimal hyperplane as the domain of novelty. The margin, i.e., the distance from the origin to the optimal hyperplane, is maximized to make the contours enclosing the data as tightly as possible. We subsequently apply the stochastic gradient descent framework proposed in [19] to the first phase of support-based clustering for achieving the domain of novelty. Finally, we propose a new strategy for clustering assignment where each data sample in the extended decision boundary has its own trajectory to converge to an equilibrium point and clustering assignment is then reduced to the same task for those equilibrium points. Our clustering assignment strategy distinguishes from the existing works of [8, 11, 12, 13] in the way to find the trajectory with a start and the initial set of data samples that need to do a trajectory for finding the corresponding equilibrium point. The experiments established on the real-world datasets show that our proposed method produces the comparable clustering quality with other support-based clustering methods while simultaneously achieving the computational speedup.

After solving the optimization problem, we yield the decision function

$$\begin{aligned} f\left( x\right) =\sum _{i=1}^{N}\alpha _{i}K\left( x_{i},x\right) -1 \end{aligned}$$

To find the equilibrium points, we need to solve the equation \(\nabla f\left( x\right) =0\). To this end, we use the fixed point technique and assume that Gaussian kernel is used, i.e., \(K\left( x,x'\right) =e^{-\gamma \Vert x-x'\Vert ^{2}}\). We then have

$$\begin{aligned} \frac{1}{2}\nabla f\left( x\right)= & {} \sum _{i=1}^{N}\alpha _{i}\left( x_{i}-x\right) e^{-\gamma \Vert x-x_{i}\Vert ^{2}} \\= & {} 0\rightarrow x=\frac{\sum _{i=1}^{N}\alpha _{i}e^{-\gamma \Vert x-x_{i}\Vert ^{2}}x_{i}}{\sum _{i=1}^{N}\alpha _{i}e^{-\gamma \Vert x-x_{i}\Vert ^{2}}}=P\left( x\right) \end{aligned}$$

To find an equilibrium point, we start with the initial point \(x^{(0)}\in \mathbb {R}^{d}\) and iterate \(x^{(j+1)}=P\left( x^{(j)}\right) \). By fixed point theorem, the sequence \(x^{(j)}\), which can be considered as a trajectory with start \(x^{(0)}\), converges to the point \(x_{*}^{(0)}\) satisfying P\((x_{*}^{(0)})=x_{*}^{(0)}\) or \(\nabla f(x_{*}^{(0)})=0\), i.e., \(x_{*}^{(0)}\) is an equilibrium point.

Open image in new window

Let us denote \(B_{\epsilon }=\left\{ x_{i}:1\le i\le N\,\wedge \,\vert f\left( x_{i}\right) \vert \le \epsilon \right\} \), namely the extended boundary for a tolerance \(\epsilon >0\). It follows that the set \(B_{\epsilon }\) forms a strip enclosing the decision boundary \(f\left( x\right) =0\). Algorithm 2 is proposed to do clustering assignment. In Algorithm 2, the task of clustering assignment is reduced to itself for M equilibrium point. To fulfill cluster assignment for M equilibrium points, we run \(m=20\) sample-point test as proposed in [1].

Open image in new window

Our proposed clustering assignment procedure is different with the existing procedure proposed in [1]. The procedure proposed in [1] requires to run \(m=20\) sample-point test for every edge connected \(x_{i},\,x_{j}\) (\(i\ne j\)) in the training set. Consequently, the computational cost incurred is \(\text {O}\left( N\left( N-1\right) ms\right) \) where s is the sparsity level of the decision function (i.e., the number of vectors in the model). Our proposed procedure needs to perform \(m=20\) sample-point test for a reduced set of M data samples (i.e., the set of the equilibrium points \(\left\{ e_{1},e_{2},\ldots ,e_{M}\right\} \)) where M is possibly very small comparing with N. The reason is that many data points in the training set could converge to a common equilibrium point which significantly reduces the size from N to M. The computational cost incurred is therefore \(\text {O}\left( M\left( M-1\right) ms\right) \).

In this paper, we have proposed a fast support-based clustering method, which conjoins the advantages of SGD-based method and kernel-based method. Furthermore, we have also proposed a new strategy for clustering assignment. We validate our proposed method on 15 well-known datasets for clustering task. The experiment has shown that our proposed method has achieved a comparable clustering quality compared with the baselines while being significantly faster than them.