Discriminative clustering with representation learning with any ratio of labeled to unlabeled data

Abstract

We present a discriminative clustering approach in which the feature representation can be learned from data and moreover leverage labeled data. Representation learning can give a similarity-based clustering method the ability to automatically adapt to an underlying, yet hidden, geometric structure of the data. The proposed approach augments the DIFFRAC method with a representation learning capability, using a gradient-based stochastic training algorithm and an optimal transport algorithm with entropic regularization to perform the cluster assignment step. The resulting method is evaluated on several real datasets when varying the ratio of labeled data to unlabeled data and thereby interpolating between the fully unsupervised regime and the fully supervised regime. The experimental results suggest that the proposed method can learn powerful feature representations even in the fully unsupervised regime and can leverage even small amounts of labeled data to improve the feature representations and to obtain better clusterings of complex datasets.

Appendices

Appendix

Smoothness of the objective function

In this appendix we estimate the smoothness constants for “forward prediction” regularized least squares and “reverse prediction” least squares, following the terminology of Section 3 of Xu et al. (2009). Regularized forward prediction least squares learns to predict the label matrix Y from the features $\Phi$:

$$\begin{aligned} \min _W \frac{1}{n}\Vert Y-\Phi W -\mathbb {1}_n b^T\Vert _F^2+\lambda \Vert W\Vert ^2_F\;. \end{aligned}$$

In contrast, reverse prediction least squares learns to predict the features $\Phi$ from the labels Y:

$$\begin{aligned} \min _{W}\frac{1}{n}\Vert \Phi -YW\Vert _F^2\,. \end{aligned}$$

As noted by Xu et al. (2009), the solution of the forward prediction problem can be recovered from the solution of the reverse prediction problem as long as $\Phi$ is full rank.

Now we return to Proposition 1, which compared the Lipschitz constants of the two objectives, and provide its proof.

Proof

After minimizing in the classifier variable W, the forward prediction objective reads

$$\begin{aligned} F_f(\Phi ) = \lambda {{\,\mathrm{tr}\,}}[YY^T\Pi _n(\Pi _n\Phi \Phi ^T\Pi _n + n\lambda {\text { I}})^{-1}\Pi _n]\,. \end{aligned}$$

Define $G(\Phi ) = \left( \Pi _n\Phi \Phi ^T\Pi _n+ n\lambda {\text { I}}_{n}\right) ^{-1}$. The gradient of $F_f$ is then

$$\begin{aligned} \nabla F_f(\Phi ) = -2\lambda \Pi _nG(\Phi ) \Pi _nYY^T\Pi _n G(\Phi ) \Pi _n\Phi . \end{aligned}$$

Since $\Vert G(\Phi )\Vert _2\le 1/(n\lambda )$, $\Vert YY^T\Vert _2\le n_\text {max}$, $\Vert \Pi _n\Vert _2 \le 1$ and $\Vert G(\Phi ) \Pi _n\Phi \Vert _2\le {\Vert \Pi _n\Phi \Vert _2}/{(n \lambda )}$, we obtain

$$\begin{aligned} \Vert \nabla F_f(\Phi ) \Vert _2&\le \frac{2Bn_{\max }}{n^2\lambda } {=}{:}L_f\,. \end{aligned}$$

Recall that the reverse prediction objective for fixed cluster assignments Y may be written as $F_r(\Phi )\! =\! \frac{1}{n}{{\,\mathrm{tr}\,}}[({\text { I}}-P_Y)\Phi \Phi ^T]$ where $P_Y=Y(Y^TY)^{-1}Y^T$ is an orthonormal projector. Its gradient, $\nabla F_r(\Phi ) = \frac{2}{n}({\text { I}}-P_Y)\Phi$, can therefore be bounded as

$$\begin{aligned} \Vert \nabla F_r(\Phi )\Vert _2 \le 2B/n {=}{:}L_r\,. \end{aligned}$$

Hence, taking $\lambda \ge n_{\max }/n$, we have $L_f\le L_r$. $\square$

Before moving on to the smoothness of the gradient we prove a lemma. The lemma estimates the Lipschitz constant of the gradient of the “forward prediction” objective function $F_f(\Phi )$ from Sect. 4.1.

Lemma 1

Consider a feature matrix $\Phi \in \mathbb {R}^{n\times D}$ from the set of all possible feature matrices $\mathcal {Z}$. Define the function $F_f:\mathbb {R}^{n\times D}\rightarrow \mathbb {R}$ by $F_f(\Phi ) = \lambda {{\,\mathrm{tr}\,}}[YY^T\Pi _n(\Pi _n \Phi \Phi ^T\Pi _n + n\lambda {\text { I}})^{-1}\Pi _n].$ Assume there exists B such that for all $\Phi \in \mathcal {Z}$, $\Vert \Phi \Vert _2\le B$. Then for all $\Phi _1,\Phi _2\in \mathcal {Z}$,

$$\begin{aligned}&\left\| \nabla F_f(\Phi _1)-\nabla F_f(\Phi _2)\right\| _2 \\ \le&\left( \frac{8B^2n_{\max }}{n^3\lambda ^2} + \frac{2n_{\max }}{n^2\lambda }\right) \Vert \Phi _1-\Phi _2\Vert _2\,. \end{aligned}$$

Hence, an upper bound on the Lipschitz constant of the gradient of $F_f(\Phi )$ is given by

$$\begin{aligned} \ell _f {:}{=}\frac{2n_{\max }}{n^2\lambda }+\frac{8B^2n_{\max }}{n^3\lambda ^2} \,. \end{aligned}$$

Proof

Note that the gradient of $F_f$ is given by $\nabla F_f(\Phi ) = -2\lambda \Pi _n G(\Phi ) \Pi _nYY^T\Pi _n G(\Phi ) \Pi _n\Phi$, where we have defined $G(\Phi )=(\Pi _n\Phi \Phi ^T\Pi _n+n\lambda I)^{-1}$. Now define ${\tilde{Y}} = \Pi _nY$, ${\tilde{\Phi }}_1= \Pi _n\Phi _1$, and ${\tilde{\Phi }}_2= \Pi _n\Phi _2$. Moreover, let $\Vert \cdot \Vert =\Vert \cdot \Vert _2$ denote the spectral norm. Using the fact that $\Vert \Pi _n\Vert \le 1$ since $\Pi _n$ is a projection matrix, observe that

$$\begin{aligned}&\frac{1}{2\lambda }\left\| \nabla F_f(\Phi _1)-\nabla F_f(\Phi _2)\right\| \\ =\,&\left\| \Pi _n \left[ G(\Phi _1){\tilde{Y}}{\tilde{Y}}^TG(\Phi _1){\tilde{\Phi }}_1-G(\Phi _2){\tilde{Y}}{\tilde{Y}}^T G(\Phi _2){\tilde{\Phi }}_2\right] \right\| \\ \le \,&\underbrace{\left\| G(\Phi _1){\tilde{Y}}{\tilde{Y}}^TG(\Phi _1){\tilde{\Phi }}_1-G(\Phi _2){\tilde{Y}}{\tilde{Y}}^TG(\Phi _2){\tilde{\Phi }}_1 \right\| }_{(a)} \\&+ \underbrace{\left\| G(\Phi _2){\tilde{Y}}\tilde{Y}^TG(\Phi _2){\tilde{\Phi }}_1-G(\Phi _2){\tilde{Y}}{\tilde{Y}}^T G(\Phi _2){\tilde{\Phi }}_2 \right\| }_{(b)}\,. \end{aligned}$$

First consider term (a). We have that

$$\begin{aligned}&\left\| G(\Phi _1){\tilde{Y}}{\tilde{Y}}^TG(\Phi _1){\tilde{\Phi }}_1-G(\Phi _2){\tilde{Y}}{\tilde{Y}}^TG(\Phi _2){\tilde{\Phi }}_1 \right\| \\ \le \,&\underbrace{\left\| G(\Phi _1){\tilde{Y}}{\tilde{Y}}^TG(\Phi _1)-G(\Phi _2){\tilde{Y}}{\tilde{Y}}^TG(\Phi _2) \right\| }_{(c)} \Vert \Phi _1\Vert \,. \end{aligned}$$

We may bound term (c) by

$$\begin{aligned}&\left\| G(\Phi _1){\tilde{Y}}{\tilde{Y}}^TG(\Phi _1)-G(\Phi _2){\tilde{Y}}{\tilde{Y}}^TG(\Phi _2) \right\| \\&\quad \le \, \left\| G(\Phi _1){\tilde{Y}}{\tilde{Y}}^TG(\Phi _1)-G(\Phi _1){\tilde{Y}}{\tilde{Y}}^TG(\Phi _2) \right\| \\&\qquad +\left\| G(\Phi _1){\tilde{Y}}{\tilde{Y}}^TG(\Phi _2)-G(\Phi _2){\tilde{Y}}{\tilde{Y}}^T G(\Phi _2) \right\| \\&\quad \le \, \left\| G(\Phi _1){\tilde{Y}}{\tilde{Y}}^T\right\| \underbrace{\left\| G(\Phi _1)-G(\Phi _2) \right\| }_{(d)} \\&\qquad + \left\| G(\Phi _2){\tilde{Y}}{\tilde{Y}}^T\right\| \left\| G(\Phi _1)-G(\Phi _2) \right\| \,. \end{aligned}$$

Furthermore, we can bound term (d) via

$$\begin{aligned}&\left\| G(\Phi _1)-G(\Phi _2) \right\| \\&\quad =\, \left\| G(\Phi _1)\left[ G(\Phi _1)^{-1}-G(\Phi _2)^{-1}\right] G(\Phi _2) \right\| \\&\quad \le \, \left\| G(\Phi _1)\right\| \left\| G(\Phi _2)\right\| \underbrace{\left\| G(\Phi _1)^{-1}-G(\Phi _2)^{-1}\right\| }_{(e)} \;. \end{aligned}$$

Finally, we can bound term (e) using

$$\begin{aligned}&\left\| G(\Phi _1)^{-1}-G(\Phi _2)^{-1}\right\| \\&\quad =\, \left\| \Pi _n \Phi _1\Phi _1^T \Pi _n- \Pi _n\Phi _2\Phi _2^T \Pi _n\right\| \\&\quad \le \, \left\| \Phi _1\Phi _1^T-\Phi _1\Phi _2^T\right\| + \left\| \Phi _1\Phi _2^T-\Phi _2\Phi _2^T\right\| \\&\quad \le \, \Vert \Phi _1\Vert \Vert \Phi _1-\Phi _2\Vert + \Vert \Phi _2\Vert \Vert \Phi _1-\Phi _2\Vert \,. \end{aligned}$$

Using this above, a bound on term (a) is thus

$$\begin{aligned}&\left\| G(\Phi _1){\tilde{Y}}{\tilde{Y}}^TG(\Phi _1){\tilde{\Phi }}_1-G(\Phi _2){\tilde{Y}}{\tilde{Y}}^TG(\Phi _2){\tilde{\Phi }}_1 \right\| \\&\quad \le \, \left( \left\| G(\Phi _1){\tilde{Y}}{\tilde{Y}}^T\right\| + \left\| G(\Phi _2){\tilde{Y}}{\tilde{Y}}^T\right\| \right) \left\| G(\Phi _1)\right\| \left\| G(\Phi _2)\right\| \\&\qquad \times \left( \Vert \Phi _1\Vert +\Vert \Phi _2\Vert \right) \Vert \Phi _1\Vert \Vert \Phi _1-\Phi _2\Vert \,. \end{aligned}$$

Next, consider term (b). We have that

$$\begin{aligned}&\left\| G(\Phi _2){\tilde{Y}}{\tilde{Y}}^TG(\Phi _2){\tilde{\Phi }}_1-G(\Phi _2){\tilde{Y}}{\tilde{Y}}^T G(\Phi _2){\tilde{\Phi }}_2 \right\| \\&\quad \le \, \left\| G(\Phi _2){\tilde{Y}}{\tilde{Y}}^TG(\Phi _2) \right\| \left\| \Phi _1-\Phi _2\right\| \,. \end{aligned}$$

Therefore, returning to the original quantity of interest, we have

$$\begin{aligned}&\frac{1}{2\lambda }\left\| \nabla F_f(\Phi _1)-\nabla F_f(\Phi _2)\right\| \\&\quad \le \, \left\{ \left( \left\| G(\Phi _1){\tilde{Y}}{\tilde{Y}}^T\right\| + \left\| G(\Phi _2){\tilde{Y}}{\tilde{Y}}^T\right\| \right) \left\| G(\Phi _1)\right\| \left\| G(\Phi _2)\right\| \right. \\&\qquad \left. \times \left( \Vert \Phi _1\Vert +\Vert \Phi _2\Vert \right) \Vert \Phi _1\Vert + \left\| G(\Phi _2){\tilde{Y}}{\tilde{Y}}^TG(\Phi _2) \right\| \right\} \\&\qquad \times \Vert \Phi _1-\Phi _2\Vert \,. \end{aligned}$$

Next, note that $\Vert YY^T\Vert _2\le n_{\max }$, where $n_{\max }$ is a bound on the maximum size of the clusters. Lastly, $\Vert G(\Phi _1)\Vert _2\le 1/(n\lambda )$ and $\Vert G(\Phi _2)\Vert _2\le 1/(n\lambda )$. Therefore, we have

$$\begin{aligned}&\frac{1}{2\lambda }\left\| \nabla F_f(\Phi _1)-\nabla F_f(\Phi _2)\right\| _2\\&\quad \le \, \left( \frac{4B^2n_{\max }}{n^3\lambda ^3} + \frac{n_{\max }}{n^2\lambda ^2}\right) \Vert \Phi _1-\Phi _2\Vert _2\,, \end{aligned}$$

and so an upper bound on the Lipschitz constant is given by

$$\begin{aligned} \ell _f {:}{=}\frac{2n_{\max }}{n^2\lambda }+\frac{8B^2n_{\max }}{n^3\lambda ^2} \,. \end{aligned}$$

$\square$

Recall that Proposition 2 from Sect. 4.1, compared the Lipschitz constants of the gradients of the forward and reverse prediction objectives. Here we provide its proof.

Proof

The gradient of $F_f$ is given by

$$\begin{aligned} \nabla F_f(\Phi ) = -2\lambda \Pi _nG(\Phi ) \Pi _n Y \tilde{Y}^T \Pi _n G(\Phi ) \Pi _n \Phi \,. \end{aligned}$$

By Lemma 1 we have that

$$\begin{aligned}&\left\| \nabla F_f(\Phi _1)-\nabla F_f(\Phi _2)\right\| _2\\&\quad \le \left( \frac{2n_{\max }}{n^2\lambda } + \frac{8B^2n_{\max }}{n^3\lambda ^2}\right) \Vert \Phi _1-\Phi _2\Vert _2\,. \end{aligned}$$

Next, observe that the gradient of $F_r$ is

$$\begin{aligned} \nabla F_r(\Phi ) = \frac{2}{n}({\text { I}}-P_Y)\Phi \,. \end{aligned}$$

Hence, we have

$$\begin{aligned} \Vert \nabla F_r(\Phi _1)-F_r(\Phi _2)\Vert _2 \le \frac{2}{n}\Vert \Phi _1-\Phi _2\Vert _2\,. \end{aligned}$$

For $\lambda \ge n_{\max }/(2n)+\sqrt{n_{\max }^2+16B^2n_{\max }}/(2n)$, we therefore have $\ell _f\le \ell _r$. $\square$

NP-completeness of the label assignment problem

Now we address the problem of optimizing the labels for the unlabeled data. The following proposition shows that this discrete problem is in general NP-complete for $k>2$.

Proposition 3

Let $A\in \mathbb {R}^{n\times n}$. The label assignment problem

$$\begin{aligned} \min _Y\,&{{\,\mathrm{tr}\,}}(YY^TA)\\ s.t.\,&\sum _{j=1}^k Y_{ij}=1, \quad i=1,\dots , n\\&Y_{ij}\in \{0,1\} \quad \forall \, i=1,\dots , n,\, j=1,\dots , k \end{aligned}$$

is NP-complete for $k>2$.

Proof

The proof will follow by showing that the k-coloring problem is a special case of the matrix balancing problem. Let G be an undirected, unweighted graph with no self-loops. Define $A\in \{0,1\}^{n\times n}$ to be the adjacency matrix of G. Then G is k-colorable if and only if the following problem has minimum value zero:

$$\begin{aligned} \min _Y\,&\sum _{j=1}^k \sum _{i,i'\in A} Y_{i,j}Y_{i',j} \\ s.t.\,&\sum _{j=1}^k Y_{ij}=1, \quad i=1,\dots , n\\&Y_{ij}\in \{0,1\} \quad \forall \, i=1,\dots , n, \, j=1,\dots , k\,. \end{aligned}$$

Noting that

$$\begin{aligned} \sum _{j=1}^k \sum _{i,i'\in A} Y_{i,j}Y_{i',j} = {{\,\mathrm{tr}\,}}(YY^TA)\,, \end{aligned}$$

we may rewrite the above problem as

$$\begin{aligned} \min _Y\,&{{\,\mathrm{tr}\,}}(YY^TA) \\ s.t.\,&\sum _{j=1}^k Y_{ij}=1, \quad \forall \, i=1,\dots , n\\&Y_{ij}\in \{0,1\} \quad \forall \, i=1,\dots , n,\, j=1,\dots , k\,. \end{aligned}$$

This is a special case of the matrix balancing problem, in which A is the adjacency matrix of a graph. Therefore, as the k-coloring problem is NP-complete for $k>2$ (Karp 1975), the label assignment problem with discrete assignments is also NP-complete for $k>2$. $\square$

An alternative relaxation

Bach and Harchaoui (2007) propose alternative relaxations of the labeling subproblem. Define $\lambda _1 \le \lambda _2 \le \cdots \le \lambda _n$ to be the eigenvalues of the equivalence matrix M and let $\lambda _0\ge 0$. In Section 2.6 of their paper Bach and Harchaoui suggest solving the problem

$$\begin{aligned} \min _{M \in \mathbb {R}^{n \times n}} \quad&{{\,\mathrm{tr}\,}}(M^T A) \nonumber \\ \text{ subject } \text{ to } \quad&M=M^T \nonumber \\&{{\,\mathrm{tr}\,}}(M) = n \nonumber \\&M \succeq 0 \nonumber \\&\sum _{i=1}^n \min \left\{ \frac{\lambda _i}{\lambda _0}, 1\right\} \ge k \end{aligned}$$

(6)

in the unsupervised setting.

1.1 Derivation of the solution

Note that the symmetric and positive semi-definite constraints imply that we can write $M=U\Lambda U^T$ where U contains an orthonormal set of eigenvectors of M and $\Lambda \ge 0$ is a diagonal matrix containing the corresponding eigenvalues. After rewriting ${{\,\mathrm{tr}\,}}(M^T A)=\sum _{i=1}^n \lambda _iu_i^TAu_i$, with $u_i = U_{\cdot , i}$, we obtain the problem

$$\begin{aligned} \min _{U \in \mathbb {R}^{n \times n}, \lambda _1,\dots , \lambda _n\in \mathbb {R}} \quad&\sum _{i=1}^n \lambda _iu_i^TAu_i \\ \text{ subject } \text{ to } \quad&\sum _{i=1}^n \lambda _i = n\\&\sum _{i=1}^n \min \left\{ \frac{\lambda _i}{\lambda _0}, 1\right\} \ge k \\&u_i^Tu_i = 1 \quad \forall i \\&u_i^Tu_j = 0 \quad \forall i\ne j \\&\lambda _i \ge 0 \quad \forall i \,. \end{aligned}$$

Introducing Lagrange multipliers and defining the Lagrangian

$$\begin{aligned}&\mathcal {L}(U, \Lambda , \alpha , \beta , \gamma , \delta , \epsilon ) \\ =\,&\sum _{i=1}^n \lambda _iu_i^TAu_i + \alpha \left( \sum _{i=1}^n \lambda _i - n\right) \\&- \beta \left( \sum _{i=1}^n \min \left\{ \frac{\lambda _i}{\lambda _0}, 1\right\} - k\right) \\&+ \sum _{i=1}^n \gamma _i(u_i^Tu_i - 1) + \sum _{i\ne j} \delta _{ij} u_i^Tu_j - \sum _{i=1}^n \epsilon _i\lambda _i \,, \end{aligned}$$

we can rewrite the problem as

$$\begin{aligned} \max _{\alpha \in \mathbb {R}, \beta \in \mathbb {R}, \gamma \in \mathbb {R}^n, \delta \in \mathbb {R}^{n^2}} \min _{U \in \mathbb {R}^{n \times n}, \lambda _1,\dots , \lambda _n\in \mathbb {R}} \quad&\mathcal {L}(U, \Lambda , \alpha , \beta , \gamma , \delta , \epsilon ) \\ \text{ subject } \text{ to } \quad&\beta \ge 0, \,\, \epsilon _i \ge 0 \,\,\, \forall i\,, \end{aligned}$$

where $\alpha \in \mathbb {R}, \beta \in \mathbb {R}, \gamma \in \mathbb {R}^n, \delta \in \mathbb {R}^{n^2}, \epsilon \in \mathbb {R}^n$ and we define $\delta _{ii}=0$ for all i. The optimal parameter values must satisfy the first order conditions

$$\begin{aligned}&2\lambda _i^\star Au_i^\star + 2\gamma _i^\star u_i^\star + \sum _{i\ne j} \delta _{ij}^\star u_j^\star = 0 \quad \forall i \nonumber \\&{u_i^\star }^TAu_i^\star + \alpha ^\star - \beta ^\star \left[ \frac{1}{2\lambda _0}(1-\text {sign}(\lambda _i^\star -\lambda _0))\right] - \epsilon _i^\star \ni 0 \quad \forall i \,. \end{aligned}$$

(7)

From line (7) we can see that $UAU^T$ is diagonal, and hence U consists of a set of eigenvectors of A. Defining $0 \le a_1\le a_2\le \cdots \le a_n$ to be the eigenvalues of A, we can then rewrite the problem as

$$\begin{aligned} \min _{\lambda _1,\dots , \lambda _n\in \mathbb {R}} \quad&\sum _{i=1}^n \lambda _ia_i \end{aligned}$$

(8)

$$\begin{aligned} \text { subject } \text { to } \quad&\sum _{i=1}^n \lambda _i = n \nonumber \\&\sum _{i=1}^n \min \left\{ \frac{\lambda _i}{\lambda _0}, 1\right\} \ge k \nonumber \\&\lambda _i \ge 0 \quad \forall i \,. \end{aligned}$$

(9)

To solve this, consider a possible solution ${\tilde{\lambda }}_1,\dots , {\tilde{\lambda }}_n$. We will consider several cases. First, suppose there exists $i < j$ such that ${\tilde{\lambda }}_i, {\tilde{\lambda }}_j > \lambda _0$. Then define ${\tilde{\lambda }}_i'= {\tilde{\lambda }}_i + {\tilde{\lambda }}_j-\lambda _0$, ${\tilde{\lambda }}_j'=\lambda _0$, and ${\tilde{\lambda }}_m'={\tilde{\lambda }}_m$ for $m\notin \{i,j\}$. Since ${\tilde{\lambda }}_i', {\tilde{\lambda }}_j' \ge \lambda _0$ and ${\tilde{\lambda }}_i'+{\tilde{\lambda }}_j'={\tilde{\lambda }}_i+{\tilde{\lambda }}_j$ the constraints are still satisfied. Therefore, since $a_i\le a_j$, $\sum _{i=1}^n {\tilde{\lambda }}_i'a_i \le \sum _{i=1}^n {\tilde{\lambda }}_ia_i$, and so we know that there always exists an optimum with at most one i such that $\lambda _i>\lambda _0$. Moreover, suppose that this index i is larger than 1. Then, we could set ${\tilde{\lambda }}_i'= {\tilde{\lambda }}_1$, ${\tilde{\lambda }}_1'={\tilde{\lambda }}_i$, and $\tilde{\lambda }_m'={\tilde{\lambda }}_m$ for $m\notin \{1,i\}$, thereby obtaining $\sum _{i=1}^n {\tilde{\lambda }}_i'a_i \le \sum _{i=1}^n {\tilde{\lambda }}_ia_i$. Thus, there always exists an optimum $\lambda _1^\star ,\dots , \lambda _n^\star$ with $\lambda _2^\star ,\dots , \lambda _n^\star \le \lambda _0.$

Next, suppose there exists $i<j$ such that $0<{\tilde{\lambda }}_i, {\tilde{\lambda }}_j < \lambda _0$. Then define $\tilde{\lambda }_i'={\tilde{\lambda }}_i + \min \{\lambda _0-{\tilde{\lambda }}_i, {\tilde{\lambda }}_j\}$, ${\tilde{\lambda }}_j'={\tilde{\lambda }}_j-\min \{\lambda _0-{\tilde{\lambda }}_i, {\tilde{\lambda }}_j\}$, and ${\tilde{\lambda }}_m'={\tilde{\lambda }}_m$ for $m\notin \{i,j\}$. Since ${\tilde{\lambda }}_i', {\tilde{\lambda }}_j' \le \lambda _0$ and ${\tilde{\lambda }}_i'+{\tilde{\lambda }}_j'={\tilde{\lambda }}_i+{\tilde{\lambda }}_j$ the constraints are still satisfied. Therefore, since $a_i\le a_j$, $\sum _{i=1}^n {\tilde{\lambda }}_i'a_i \le \sum _{i=1}^n {\tilde{\lambda }}_ia_i$, we know that there always exists an optimum with at most one i such that $0<\lambda _i<\lambda _0$. Now suppose that this i is not the largest index such that $\lambda _i>0$. Then there exists an optimum with a $j>i$ such that $\lambda _j=\lambda _0$. Then we could set ${\tilde{\lambda }}_i'= \tilde{\lambda }_j$, ${\tilde{\lambda }}_j'={\tilde{\lambda }}_i$, and $\tilde{\lambda }_m'={\tilde{\lambda }}_m$ for $m\notin \{i,j\}$, thereby obtaining $\sum _{i=1}^n {\tilde{\lambda }}_i'a_i \le \sum _{i=1}^n {\tilde{\lambda }}_ia_i$. Thus, there always exists an optimum $\lambda _1^\star ,\dots , \lambda _n^\star$ with $\lambda _1^\star \ge \lambda _0$, $\lambda _2^\star ,\dots , \lambda _{i-1}^\star =\lambda _0$, $0\le \lambda _i^\star \le \lambda _0$ for some i, and, if $i\ne n$, $\lambda _{i+1}^\star ,\dots , \lambda _n^\star =0$.

Now from constraint (9) we can see that there must exist at least k non-zero $\lambda _i$’s in the solution. If $n=k$, then we must have $\lambda _0=1$ and hence the optimum is given by $\lambda _1^\star ,\dots , \lambda _k^\star =1$. Now consider the case where $n>k$. Suppose there exists a solution ${\tilde{\lambda }}_1,\dots , {\tilde{\lambda }}_n$ such that ${\tilde{\lambda }}_{k+1} \ne 0$. Then, since ${\tilde{\lambda }}_1,\dots , {\tilde{\lambda }}_k \ge \lambda _0$ we can set ${\tilde{\lambda }}_1'={\tilde{\lambda }}_1+{\tilde{\lambda }}_{k+1}$, ${\tilde{\lambda }}_{k+1} = 0$, and ${\tilde{\lambda }}_j'={\tilde{\lambda }}_j$ for $j\notin \{1,k+1\}$. This once again satisfies the constraints and $\sum _{i=1}^n {\tilde{\lambda }}_i'a_i \le \sum _{i=1}^n {\tilde{\lambda }}_ia_i$. Therefore, there exists a solution such that $\lambda _1\ge \lambda _0$ and $\lambda _2,\dots , \lambda _k=\lambda _0$. In particular, a solution is $\lambda _1^\star =n-(k-1)\lambda _0$, $\lambda _2^\star ,\dots , \lambda _k^\star =\lambda _0$.

In summary, the optima of this problem depend on the values of k and n. In particular, we have:

• If $n>k$, an optimum is given by $\lambda _1^\star =n-(k-1)\lambda _0$, $\lambda _2^\star ,\dots , \lambda _k^\star =\lambda _0$, $\lambda _{k+1}^\star ,\dots , \lambda _n^\star =0$.

• If $n=k$, the optimum is given by $\lambda _1^\star ,\dots , \lambda _k^\star =1$.

Returning to the original problem (6), we therefore have that an optimal M is

$$\begin{aligned} M^\star =\sum _{i=1}^n \lambda _i^\star u_iu_i^T\,, \end{aligned}$$

where $u_1,\dots , u_n$ are eigenvectors corresponding to the eigenvalues $a_1\le a_2,\dots \le a_n$ of A and where $\lambda _1^\star ,\dots , \lambda _n^\star$ are as defined above.

1.2 Comparison to the XSDC relaxation

We now compare the convex relaxation of the labeling subproblem presented in Sect. 4.2 to the relaxation proposed by Bach and Harchaoui (2007). As accommodating constraints on cluster labels is less natural in the latter relaxation, we compare the relaxations when training a LeNet-5 CKN on MNIST with no labeled data. Figure 12 compares our matrix balancing method, the eigendecomposition method from the previous subsection, and the eigendecomposition method followed by k-means clustering. Prior to clustering, the rows of the eigenvector matrix were normalized to have unit $\ell _2$ norm. The value of $\lambda _0$ was chosen from the set $\{0.01n_b, 0.02n_b,\dots , 0.1n_b\}$, where $n_b$ is the size of a mini-batch, based on the performance on the validation set.

Fig. 12

Performance of matrix balancing in comparison to eigendecomposition-based methods across 10 trials of training a LeNet-5 CKN on MNIST with no labeled data

Fig. 13

Evolution of the eigengap from matrix balancing in comparison to eigendecomposition-based methods across 10 trials of training a LeNet-5 CKN on MNIST with no labeled data. The error bands show one standard deviation from the mean

From Fig. 12 we can see that the convex relaxation used to derive the matrix balancing method is superior to the relaxations leading to the eigendecomposition-based methods. On average, matrix balancing performs 17% better than the eigendecomposition method and 12% better than the eigendecomposition method followed by k-means. This suggests that the constraint from the convex relaxation requiring the diagonal of M to consist of all 1’s and/or the constraint requiring all entries of M to be positive are important for the performance of the labeling method.

In Fig. 13 we examine the eigengap of A across iterations. The eigengap is defined as $\lambda _{k+1}-\lambda _{k}$, where k is the number of classes and $\lambda _1\le \dots \le \lambda _n$ are the eigenvalues of A. As noted by Meila et al. (2005), having a larger eigengap makes the subspace spanned by the first k eigenvectors of A more stable to perturbations. From the figure we can see that the eigendecomposition-based methods tend to increase the eigengap as the learning proceeds. For the eigendecomposition method, the eigengap increased from $2\times 10^{-6}$ to $6\times 10^{-5}$ on average after 50 iterations. Similarly, for the eigendecomposition method followed by k-means, the eigengap increased from $5\times 10^{-6}$ to $5\times 10^{-5}$ on average after 50 iterations. It is interesting to note that matrix balancing, which does not yield low-rank solutions $M^\star$, leads to eigengaps that are extremely small (on the order of $10^{-15}$) across the iterations. Nevertheless, it outperforms the eigendecomposition-based methods.

Additional experimental details

Here, we provide additional details related to the training and the additional constraints we consider.

1.1 Parameter tuning

The algorithm proposed in this paper and the models used require a large number of parameters to be set. Next we discuss the choices for these parameters.

1.1.1 Fixed parameters

The parameters that are fixed throughout the experiments and not validated are as follows. The number of filters in the networks is set to 32 and the network’s parameters V are initialized layer-wise with 32 feature maps drawn uniformly at random from the output of the previous layer. The networks use the Nyström method to approximate the kernel at each layer. The regularization in the Nyström approximation is set to 0.001, and 20 Newton iterations are used to compute the inverse square root of the Gram matrix on the parameters $V_\ell$ at each layer $\ell$, as done by Jones (2020). The bandwidth is set to the median pairwise distance between the first 1000 observations for the single-layer networks. It is set to 0.6 for the convolutional networks. The batch size for both the labeled and unlabeled data is set to 4096 for Gisette and MAGIC and 1024 for MNIST and CIFAR-10 (due to GPU memory constraints). The features output by the network $\phi$ are centered and normalized so that on average they have unit $\ell _2$ norm, as in Mairal et al. (2014). The initial training phase on just the labeled data is performed for 100 iterations, as the validation loss has typically started leveling off by 100 iterations. The entropic regularization parameter $\nu$ in the matrix balancing is set to the median absolute value of the entries in A. If this value results in divergence of the algorithm, it is multiplied by a factor of two until the algorithm converges. The value $n_\Delta$ is set to zero unless otherwise specified. The number of iterations of alternating minimization in the matrix balancing algorithm is set to 10. The number of nearest neighbors used for estimating the labels on the unlabeled data is set to 1.

1.1.2 Hold-out validation

Due to the large number of hyperparameters, we tune them sequentially as follows when labeled data, and hence a labeled validation set, exists. First, we tune the penalty $\lambda$ on the classifier weights over the values $2^i$ for $i=-40,-39,\dots ,0$. To do so, we train the classifier on only the labeled data using the initial random network parameters. We then re-validate this value every 100 iterations. Next, we tune the learning rate for the labeled data. For the modest value of $\zeta =2^{-4}$ we validate the fixed learning rate for the labeled data over the values $2^i$ for $i=-10,-9,\dots , 5$. To evaluate the performance the labels for the unlabeled data are estimated using 1-nearest neighbor. The labeled and unlabeled data are then used to train the classifier used to compute the performance. For the unbalanced experiments on MNIST only we then tune the minimum and maximum size of the classes over the values $0.01b, 0.02b,\dots , 0.2b$, where b is the batch size (fixing the semi-supervised learning rate to $2^{-5}$). For all other experiments we fix these values to b/k, where k is the number of classes in the dataset. We then tune the semi-supervised learning rate, again over the values $2^i$ for $i=-10,-9,\dots , 5$. For the single-layer networks we then tune $\zeta$ over the values $2^i$ for $i=-10,-9,\dots , 10$. For the convolutional networks we do not penalize the filters since they are constrained to lie on the sphere.

Fig. 14

Illustration of the kinds of additional constraints that were added. Green denotes the original constraints while purple denotes the constraints that were added. The numbers outside of the grids denote the true labels

When no labeled data exists we consider the hyperparameters in the same manner as during the hold-out validation. First, we consider the values $2^i$ for $i=-10,-9,\dots , 5$ for the semi-supervised learning rate. Next we consider the values $2^i$ for $i=-40,-39,\dots ,0$ for $\lambda$. Finally, if applicable, we consider the values $2^i$ for $i=-10,-9,\dots , 10$ for $\zeta$. We report the best performance observed on the test set. Developing a method for tuning the hyperparameters on an unlabeled validation set is left for future work.

1.1.3 Comparison details

In the comparisons we substitute our matrix balancing method with alternative labeling methods and retain the remainder of the XSDC algorithm. The pseudo-labeling code is our own, but we used code from Caron et al. (2018) to implement the k-means version of deep clustering.² Two important details regarding the implementations are as follows. First, for pseudo-labeling when some of the data is labeled we estimate W and b based on the labeled data in the current mini-batch, as that is what is done in XSDC. When labeled data is not present we estimate W and b based on the cluster assignments for the entire dataset. Second, for deep clustering we modify the dimension of the dimensionality reduction. In the original implementation the authors performed PCA, reducing the dimensionality of the features output by the network to 256. As the features output by the networks we consider have dimension less than 256, we instead keep the fewest number of components that account for 95% of the variance.

We perform the parameter tuning as follows. First, we follow the tuning procedure as detailed in Sect. D.1. For pseudo-labeling there are no additional parameters to tune. However, for deep clustering there are two additional parameters to tune: the number of clusters in k-means and the number of iterations between cluster updates. During the initial parameter tuning stage these parameters are set to the true number of clusters k, and 50 iterations, respectively. Afterward we tune these two remaining parameters sequentially. We first tune the number of clusters over the values k, 2k, 4k, 8k, 16k, 32k where k is the true number of clusters. We then tune the number of iterations between cluster updates over the values 10, 25, 50, 100.

1.2 Additional constraints

In one set of experiments we examine the effect of adding additional constraints. We consider two types of constraints: (1) constraints based on knowledge of whether the label was in the set $\{4,9\}$ or not; and (2) random correct must-link and must-not-link constraints among pairs of unlabeled observations and random correct must-not-link constraints between pairs of unlabeled and labeled observations.

The two types of constraints are illustrated in Fig. 14. Each grid point (i, j), if filled, denotes whether observations i and j have the same label (1) or not (0). The true labels are the values outside of the grids. Green backgrounds correspond to knowing the labels corresponding to (i, j). Purple backgrounds denote the additional known constraints. The left-most panel gives an example of an initial matrix M in which the labels corresponding to the first two observations are known (0 and 9). The second panel shows the entries we can fill in once we know whether each observation belongs to the set $\{4,9\}$. Finally, the third panel shows random correct constraints. The constraint at entry (2, 3) is a must-not-link constraint, whereas the constraint at entry (3, 4) is a must-link constraint.