A Compressed Sensing Based Least Squares Approach to Semi-supervised Local Cluster Extraction

Abstract

A least squares semi-supervised local clustering algorithm based on the idea of compressed sensing is proposed to extract clusters from a graph with known adjacency matrix. The algorithm is based on a two-stage approach similar to the one proposed by Lai and Mckenzie (SIAM J Math Data Sci 2:368–395, 2020). However, under a weaker assumption and with less computational complexity, our algorithms are shown to be able to find a desired cluster with high probability. The “one cluster at a time" feature of our method distinguishes it from other global clustering methods. Numerical experiments are conducted on the synthetic data such as stochastic block model and real data such as MNIST, political blogs network, AT &T and YaleB human faces data sets to demonstrate the effectiveness and efficiency of our algorithms.

Appendices

Appendix A Hyperparameters for Numerical Experiments

For each cluster to be recovered, we sampled the seed vertices $\varGamma _i$ uniformly from $C_i$ during all implementations. We fix the random walk depth with $t=3$, use random walk threshold parameter $\delta =0.8$ for political blogs network and $\delta =0.6$ for all the other experiments. We vary the rejection parameters $R\in (0,1)$ for each specific experiments based on the estimated sizes of clusters. In the case of no knowledge of estimated sizes of clusters nor the number of clusters are given, we may have to refer to the spectra of graph Laplacian and use the large gap between two consecutive spectrum to estimate the number of clusters, and then use the average size to estimate the size of each cluster.

We fix the least squares threshold parameter with $\gamma =0.2$ for all the experiments, which is totally heuristic. However, we have experimented that the performance of algorithms will not be affected too much by varying $\gamma \in [0.15,0.4]$. The hyperparameter MaxIter is chosen according to the size of initial seed vertices relative to the total number of vertices in the cluster. For purely comparison purpose, we keep $MaxIter=1$ for MNIST data. By experimenting on different choices of MaxIter, we find that the best performance for AT &T data occurs at $MaxIter=2$ for $10\%$ seeds and $MaxIter=1$ for $20\%$ and $30\%$ seeds. For YaleB data, the best performance occurs at $MaxIter=2$ for $5\%$, $10\%$, and $20\%$ seeds. All the numerical experiments are implemented in MATLAB and can be run on a local personal machine, for the authenticity of our results, we put a sample demo code at https://github.com/zzzzms/LeastSquareClustering the purpose of reproducibility.

Appendix B Image Data Preprocessing

For YaleB human faces data, we have performed some data preprocessing techinuqes to avoid the poor quality images. Specifically, we abandoned the pictures which are too dark, and we cropped each image into size of $54\times 46$ to reduce the effect of background noise. For the remaining qualified pictures, we randomly selected 20 images for each person.

All the image data in MNIST, AT &T, YaleB needs to be firstly constructed into an auxiliary graph before the implementation. Let $\varvec{x}_i\in \mathbb {R}^n$ be the vectorization of an image from the original data set, we define the following affinity matrix of the K-NN auxiliary graph based on Gaussian kernel according to [18, 43],

$$\begin{aligned} A_{ij} = {\left\{ \begin{array}{ll} e^{-\Vert \varvec{x}_i-\varvec{x}_j\Vert ^2/\sigma _i\sigma _j} &{} \text {if} \quad \varvec{x}_j\in NN(\varvec{x}_i,K) \\ 0 &{} \text {otherwise} \end{array}\right. } \end{aligned}$$

The notation $NN(\varvec{x}_i,K)$ indicates the set of K-nearest neighbours of $\varvec{x}_i$, and $\sigma _i:=\Vert \varvec{x}_i-\varvec{x}^{(r)}_i\Vert$ where $\varvec{x}^{(r)}_i$ is the r-th closest point of $\varvec{x}_i$. Note that the above $A_{ij}$ is not necessary symmetric, so we consider $\tilde{A}_{ij}=A^T A$ for symmetrization. Alternatively, one may also want to consider $\tilde{A}=\max \{A_{ij}, A_{ji}\}$ or $\tilde{A}=(A_{ij}+A_{ji})/2$. We use $\tilde{A}$ as the input adjacency matrix for our algorithms.

We fix the local scaling parameters $K=15$, $r=10$ for the MNIST data, $K=8$, $r=5$ for the YaleB data, and $K=5$, $r=3$ for the AT &T data during implementation.