Robust convex clustering

Abstract

Objective-based clustering is a class of important clustering analysis techniques; however, these methods are easily beset by local minima due to the non-convexity of their objective functions involved, as a result, impacting final clustering performance. Recently, a convex clustering method (CC) has been on the spot light and enjoys the global optimality and independence on the initialization. However, one of its downsides is non-robustness to data contaminated with outliers, leading to a deviation of the clustering results. In order to improve its robustness, in this paper, an outlier-aware robust convex clustering algorithm, called as RCC, is proposed. Specifically, RCC extends the CC by modeling the contaminated data as the sum of the clean data and the sparse outliers and then adding a Lasso-type regularization term to the objective of the CC to reflect the sparsity of outliers. In this way, RCC can both resist the outliers to great extent and still maintain the advantages of CC, including the convexity of the objective. Further we develop a block coordinate descent approach with the convergence guarantee and find that RCC can usually converge just in a few iterations. Finally, the effectiveness and robustness of RCC are empirically corroborated by numerical experiments on both synthetic and real datasets.

Appendix A

Hereafter we provide the proof that the objective function of the RCC in Eq. (4) is strict convex.

First, we present the definition of strict convexity as follows.

Definition 1

A function $f : {\mathbb{R}}^{n} \to {\mathbb{R}}$ is strictly convex, if ${\text{dom }}\, f$ is a convex set and if for all $\varvec{x},\varvec{ y} \in {\text{dom}} f$, $f(\theta \varvec{x} + (1 - \theta )\varvec{y}) < \theta f(\varvec{x}) + \left( {1 - \theta } \right)f(\varvec{y})$ whenever $\varvec{x} \ne \varvec{y}$ and $0 < \theta < 1$ (Boyd and Vandenberghe 2004).

Proof

As is known previously in Eq. (4), $\varvec{U} \in {\mathbb{R}}^{p \times n}$ and $\varvec{O} \in {\mathbb{R}}^{p \times n}$

Let $\tilde{\varvec{U}} = \left[ {\varvec{U},\varvec{O}} \right] \in {\mathbb{R}}^{p \times 2n}$, ${\tilde{\mathbf{v}}}_{i} = \left[ {\begin{array}{*{20}c} {{\mathbf{y}}_{n(i)} } \\ {{\mathbf{0}}_{n} } \\ \end{array} } \right] \in {\mathbb{R}}^{2n \times 1}$, ${\hat{\mathbf{v}}}_{i} = \left[ {\begin{array}{*{20}c} {{\mathbf{0}}_{n} } \\ {{\mathbf{y}}_{n(i)} } \\ \end{array} } \right] \in {\mathbb{R}}^{2n \times 1}$, ${\tilde{\mathbf{I}}} = \left[ {\begin{array}{*{20}c} {{\mathbf{I}}_{n} } \\ {{\mathbf{I}}_{n} } \\ \end{array} } \right]$.

Subsequently, the problem in Eq. (4) can be rewritten as:

$$ F_{\gamma } \left( {\tilde{\varvec{U}}} \right) = \frac{1}{2}\left\| {{\mathbf{X}} - {{\tilde{\varvec{U}}\tilde{I}}}} \right\|_{F}^{2} + \gamma_{1} \sum\nolimits_{i < j} {w_{ij} \left\| {{\tilde{\varvec{U}}(\tilde{v}}_{i} - \tilde{\varvec{v}}_{j} )} \right\|_{2} } + \gamma_{2} \sum\nolimits_{i = 1}^{n} {\left\| {{\tilde{\varvec{U}}\hat{v}}_{i} } \right\|_{1} } $$

(11)

Therefore, proving the convexity of problem (4) turns into proving that Eq. (11) is strictly convex in $\tilde{\varvec{U}}$. Applying Eq. (11) to Definition 1 for $\forall \theta \in \left( {0,1} \right)$ and assuming $\varvec{U}_{1} \ne \varvec{U}_{2}$, we further have

$$ F_{\gamma } \left( {\theta \tilde{\varvec{U}}_{1} \varvec{ + }\left( {1 - \theta } \right)\tilde{\varvec{U}}_{2} } \right) - \theta F_{\gamma } \left( {\tilde{\varvec{U}}_{1} } \right) - \left( {1 - \theta } \right)F_{\gamma } \left( {\tilde{\varvec{U}}_{2} } \right) = T_{1} + T_{2} + T_{3} $$

(12)

where

$$ \begin{aligned} T_{1} & = tr\left\{ {{\tilde{\mathbf{I}}}^{T} } \right.\left[ \theta \right.\tilde{\varvec{U}}_{1}^{T} \left( {\theta \tilde{\varvec{U}}_{1} + \left( {1 - \theta } \right)\tilde{\varvec{U}}_{2} } \right) + \left( {1 - \theta } \right)\tilde{\varvec{U}}_{2}^{T} \left( {\theta \tilde{\varvec{U}}_{1} + \left( {1 - \theta } \right)\tilde{\varvec{U}}_{2} } \right) - \left. {\theta \tilde{\varvec{U}}_{1}^{T} \tilde{\varvec{U}}_{1} - \left( {1 - \theta } \right)\tilde{\varvec{U}}_{2}^{T} \tilde{\varvec{U}}_{2} } \right]{\tilde{\mathbf{I}}} \\ T_{2} & = \gamma_{1} \sum\limits_{i < j} {w_{ij} \left\| {\left( {\theta \tilde{\varvec{U}}_{1} \varvec{ + }\left( {1 - \theta } \right)\tilde{\varvec{U}}_{2} } \right)(\tilde{\varvec{v}}_{i} - \tilde{\varvec{v}}_{j} )} \right\|_{2} } - \gamma_{1} \theta \sum\limits_{i < j} {w_{ij} \left\| {\tilde{\varvec{U}}_{1} (\tilde{\varvec{v}}_{i} - \tilde{\varvec{v}}_{j} )} \right\|_{2} } - \gamma_{1} \left( {1 - \theta } \right)\sum\limits_{i < j} {w_{ij} \left\| {\tilde{\varvec{U}}_{2} (\tilde{\varvec{v}}_{i} - \tilde{\varvec{v}}_{j} )} \right\|_{2} } \\ T_{3} & = \gamma_{2} \sum\limits_{i = 1}^{n} {\left\| {\left( {\theta \tilde{\varvec{U}}_{1} \varvec{ + }\left( {1 - \theta } \right)\tilde{\varvec{U}}_{2} } \right)\hat{\varvec{v}}_{i} } \right\|_{1} } - \gamma_{2} \theta \sum\limits_{i = 1}^{n} {\left\| {\tilde{\varvec{U}}_{1} \hat{\varvec{v}}_{i} } \right\|_{1} } - \gamma_{2} \left( {1 - \theta } \right)\sum\limits_{i = 1}^{n} {\left\| {\tilde{\varvec{U}}_{2} \hat{\varvec{v}}_{i} } \right\|_{1} } \\ \end{aligned} $$

In terms of the absolutely homogeneous and subadditive properties of the norms, T₃ in Eq. (12) is non-positive:

$$ T_{3} \le \gamma_{2} \sum\limits_{i = 1}^{n} {\left\| {\theta \tilde{\varvec{U}}_{1} \hat{\varvec{v}}_{i} } \right\|_{1} } + \gamma_{2} \sum\limits_{i = 1}^{n} {\left\| {\left( {1 - \theta } \right)\tilde{\varvec{U}}_{2} \hat{\varvec{v}}_{i} } \right\|_{1} } - \gamma_{2} \theta \sum\limits_{i = 1}^{n} {\left\| {\tilde{\varvec{U}}_{1} \hat{\varvec{v}}_{i} } \right\|_{1} } - \gamma_{2} \left( {1 - \theta } \right)\sum\limits_{i = 1}^{n} {\left\| {\tilde{\varvec{U}}_{2} \hat{\varvec{v}}_{i} } \right\|_{1} } = \gamma_{2} \theta \sum\limits_{i = 1}^{n} {\left\| {\tilde{\varvec{U}}_{1} \hat{\varvec{v}}_{i} } \right\|_{1} } + \gamma_{2} \left( {1 - \theta } \right)\sum\limits_{i = 1}^{n} {\left\| {\tilde{\varvec{U}}_{2} \hat{\varvec{v}}_{i} } \right\|_{1} } = 0 $$

Similarly, T₂ in Eq. (12) is also non-positive. Thus, Eq. (12) can be simplified to

$$ \begin{aligned} & F_{\gamma } \left( {\theta \tilde{\varvec{U}}_{1} \varvec{ + }\left( {1 - \theta } \right)\tilde{\varvec{U}}_{2} } \right) - \theta F_{\gamma } \left( {\tilde{\varvec{U}}_{1} } \right) - \left( {1 - \theta } \right)F_{\gamma } \left( {\tilde{\varvec{U}}_{2} } \right) \\ & \quad \le tr\left\{ { - \theta \left( {1 - \theta } \right){\tilde{\mathbf{I}}}^{T} } \right.\left( {\tilde{\varvec{U}}_{1} } \right. - \left. {\tilde{\varvec{U}}_{2} } \right)^{T} \left( {\tilde{\varvec{U}}_{1} } \right. - \left. {\tilde{\varvec{U}}_{2} } \right)\left. {{\tilde{\mathbf{I}}}} \right\} \\ & \quad = - \theta \left( {1 - \theta } \right)tr\left\{ {{\tilde{\mathbf{I}}}^{T} } \right.\left( {\tilde{\varvec{U}}_{1} } \right. - \left. {\tilde{\varvec{U}}_{2} } \right)^{T} \left( {\tilde{\varvec{U}}_{1} } \right. - \left. {\tilde{\varvec{U}}_{2} } \right)\left. {{\tilde{\mathbf{I}}}} \right\} \\ & \quad = - \theta \left( {1 - \theta } \right)\left\| {\left( {\tilde{\varvec{U}}_{1} } \right. - \left. {\tilde{\varvec{U}}_{2} } \right){\tilde{\mathbf{I}}}} \right\|_{F}^{2} < 0 \\ \end{aligned} $$

(.)

At last, we obtain that

$$ F_{\gamma } \left( {\theta \tilde{\varvec{U}}_{1} \varvec{ + }\left( {1 - \theta } \right)\tilde{\varvec{U}}_{2} } \right) < \theta F_{\gamma } \left( {\tilde{\varvec{U}}_{1} } \right)\varvec{ + }\left( {1 - \theta } \right)F_{\gamma } \left( {\tilde{\varvec{U}}_{2} } \right) $$

That is, $F_{\gamma } (\tilde{\varvec{U}})$ is convex in $\tilde{\varvec{U}}$. Equivalently, $F_{\gamma } (\varvec{U},\varvec{O})$ is jointly convex in $\left\{ {\varvec{U},\varvec{O}} \right\}$.