Skip to main content
SpringerLink
Log in

Self-supervised deep subspace clustering with entropy-norm

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Auto-Encoder based Deep Subspace Clustering (DSC) has been widely applied in computer vision, motion segmentation and image processing. However, existing DSC methods suffer from two limitations: (1) they ignore the rich useful relational information and the connectivity within each subspace due to the reconstruction loss; (2) they design convolutional networks individually according to specific datasets. To address the above problems and improve the performance of DSC, we propose a novel algorithm called Self-Supervised deep Subspace Clustering with Entropy-norm(S\(^{3}\)CE) in this paper. Firstly, S\(^{3}\)CE introduces self-supervised contrastive learning to pre-train the encoder instead of requiring a decoder. Besides, the trained encoder is used as a feature extractor to segment subspace by combining self-expression layer and entropy-norm constraint. This not only preserves the local structure of data, but also improves the connectivity between data points. Extensive experimental results demonstrate the superior performance of S\(^{3}\)CE in comparison to the state-of-the-art approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Vidal, René: Subspace clustering. IEEE Signal Process. Mag. 28(2), 52–68 (2011)

    Article  Google Scholar 

  2. Vidal, Rene, Ma, Yi., Sastry, Shankar: Generalized principal component analysis (gpca). IEEE Transac. Pattern Anal. Mach. Intell. 27(12), 1945–1959 (2005)

    Article  Google Scholar 

  3. Elhamifar, Ehsan, Vidal, Ren: Sparse subspace clustering: algorithm, theory, and applications. IEEE Transac. Pattern Anal. Mach. Intell. 35(11), 2765–2781 (2013)

    Article  Google Scholar 

  4. Liu, Guangcan, Lin, Zhouchen, Yong, Yu., et al.: Robust subspace segmentation by low-rank representation. In Icml 1, 8 (2010). (Citeseer)

    Google Scholar 

  5. Yang, Yingzhen, Feng, Jiashi, Jojic, Nebojsa, Yang, Jianchao, Huang, Thomas S.: \(\ell ^{0}\)-sparse subspace clustering. In Comput. Vision - ECCV 2016, 731–747 (2016)

    Google Scholar 

  6. Favaro, Paolo, Vidal, Ren, Ravichandran, Avinash: A closed form solution to robust subspace estimation and clustering. In CVPR 2011, pages 1801–1807. IEEE (2011)

  7. Lu, Canyi, Feng, Jiashi, Lin, Zhouchen, Yan, Shuicheng: Correlation adaptive subspace segmentation by trace lasso. In Proceedings of the IEEE international conference on computer vision, pages 1345–1352 (2013)

  8. Wang, Shusen, Yuan, Xiaotong, Yao, Tiansheng, Yan, Shuicheng, Shen, Jialie: Efficient subspace segmentation via quadratic programming. In Twenty-Fifth AAAI Conference on Artificial Intelligence (2011)

  9. Zhuang, Liansheng, Gao, Haoyuan, Lin, Zhouchen, Ma, Yi, Zhang, Xin, Yu, Nenghai: Non-negative low rank and sparse graph for semi-supervised learning. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2328–2335. IEEE (2012)

  10. Lu, Can-Yi, Min, Hai, Zhao, Zhong-Qiu, Zhu, Lin, Huang, De-Shuang, Yan, Shuicheng: Robust and efficient subspace segmentation via least squares regression. In European conference on computer vision, pages 347–360. Springer (2012)

  11. Zheng, Yaoguo, Zhang, Xiangrong, Yang, Shuyuan, Jiao, Licheng: Low-rank representation with local constraint for graph construction. Neurocomputing 122, 398–405 (2013)

    Article  Google Scholar 

  12. Jie Chen and Zhang Yi. Subspace clustering by exploiting a low-rank representation with a symmetric constraint. arXiv preprint arXiv:1403.2330, 2014

  13. Jiashi Feng, Zhouchen Lin, Huan Xu, and Shuicheng Yan. Robust subspace segmentation with block-diagonal prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3818–3825, 2014

  14. Xiao, Shijie, Tan, Mingkui, Dong, Xu., Dong, Zhao Yang: Robust kernel low-rank representation. IEEE Trans. Neural Netw. Learn. Syst. 27(11), 2268–2281 (2015)

    Article  MathSciNet  Google Scholar 

  15. Liang Bai and Jiye Liang. Sparse subspace clustering with entropy-norm. In International Conference on Machine Learning, pages 561–568. PMLR, 2020

  16. Xi Peng, Shijie Xiao, Jiashi Feng, Wei-Yun Yau, and Zhang Yi. Deep subspace clustering with sparsity prior. In IJCAI, pages 1925–1931, 2016

  17. Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, and Ian Reid. Deep subspace clustering networks. Advances in neural information processing systems, 30, 2017

  18. Xi Peng, Jiashi Feng, Shijie Xiao, Jiwen Lu, Zhang Yi, and Shuicheng Yan. Deep sparse subspace clustering. arXiv preprint arXiv:1709.08374, 2017

  19. Pan Zhou, Yunqing Hou, and Jiashi Feng. Deep adversarial subspace clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1596–1604, 2018

  20. Shuai Yang, Wenqi Zhu, and Yuesheng Zhu. Residual encoder-decoder network for deep subspace clustering. In 2020 IEEE International Conference on Image Processing (ICIP), pages 2895–2899. IEEE, 2020

  21. Junjian Zhang, Chun-Guang Li, Chong You, Xianbiao Qi, Honggang Zhang, Jun Guo, and Zhouchen Lin. Self-supervised convolutional subspace clustering network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5473–5482, 2019

  22. Tong Zhang, Pan Ji, Mehrtash Harandi, Wenbing Huang, and Hongdong Li. Neural collaborative subspace clustering. In International Conference on Machine Learning, pages 7384–7393. PMLR, 2019

  23. Lv, Juncheng, Kang, Zhao, Xiao, Lu., Zenglin, Xu.: Pseudo-supervised deep subspace clustering. IEEE Trans. Image Process. 30, 5252–5263 (2021)

    Article  Google Scholar 

  24. Hinton, Geoffrey E., Salakhutdinov, Ruslan R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MathSciNet  Google Scholar 

  25. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014

  26. Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013

  27. Xiaoliang Tang, Xuan Tang, Wanli Wang, Li Fang, and Xian Wei. Deep multi-view sparse subspace clustering. In Proceedings of the 2018 VII International Conference on Network, Communication and Computing, pages 115–119, 2018

  28. Abavisani, Mahdi, Patel, Vishal M.: Deep multimodal subspace clustering networks. IEEE J. Select. Topics Signal Process. 12(6), 1601–1614 (2018)

    Article  Google Scholar 

  29. Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages arXiv–1807, 2018

  30. Vidal, René, Favaro, Paolo: Low rank subspace clustering (lrsc). Pattern Recog. Lett. 43, 47–61 (2014)

    Article  Google Scholar 

  31. Pan Ji, Mathieu Salzmann, and Hongdong Li. Efficient dense subspace clustering. In IEEE Winter conference on applications of computer vision, pages 461–468. IEEE, 2014

  32. Yuzhao Ni, Ju Sun, Xiaotong Yuan, Shuicheng Yan, and Loong-Fah Cheong. Robust low-rank subspace segmentation with semidefinite guarantees. In 2010 IEEE International Conference on Data Mining Workshops, pages 1179–1188. IEEE, 2010

  33. Rahim Taheri, Meysam Ghahramani, Reza Javidan, Mohammad Shojafar, Zahra Pooranian, and Mauro Conti: Similarity-based Android malware detection using Hamming distance of static binary features. Future Gener Comput Syst. 105, 230-247. https://doi.org/10.1016/j.future.2019.11.034, 2020

  34. Peng, Zhihao, Liu, Hui, Jia, Yuheng, Hou, Junhui: Adaptive attribute and structure subspace clustering network. IEEE Trans. Image Process. 31, 3430–3439 (2022)

    Article  Google Scholar 

  35. Liu, Xiao, Zhang, Fanjin, Hou, Zhenyu, Mian, Li., Wang, Zhaoyu, Zhang, Jing: and Jie Tang. Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, Self-supervised learning (2021)

  36. Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pages 776–794. Springer, 2020

  37. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020

  38. Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020

  39. Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9640–9649, 2021

  40. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020

  41. Andreas Krause, Pietro Perona, and Ryan Gomes. Discriminative clustering by regularized information maximization. Advances in neural information processing systems, 23, 2010

  42. Man Li and Lihong Wang. Soft subspace clustering with entropy constraints. In 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), pages 920–925. IEEE, 2020

  43. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

  44. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018

  45. Zhihao Peng, Yuheng Jia, Hui Liu, Junhui Hou, and Qingfu Zhang. Maximum entropy subspace clustering network. IEEE Transactions on Circuits and Systems for Video Technology, 2021

  46. Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-20 and coil-100). 1996

  47. Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017

  48. LeCun, Yann, Bottou, Léon., Bengio, Yoshua, Haffner, Patrick: Gradient-based learning applied to document recognition. Proceed. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  49. Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

  50. M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1447–1454. IEEE, 2006

  51. Chong You, Chun-Guang Li, Daniel P Robinson, and René Vidal. Oracle based active set algorithm for scalable elastic net subspace clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3928–3937, 2016

  52. Chong You, Daniel Robinson, and René Vidal. Scalable sparse subspace clustering by orthogonal matching pursuit. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3918–3927, 2016

  53. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008

Download references

Acknowledgements

This work was supported by Public-welfare Technology Application Research of Zhejiang Province in China under Grant LGG22F020032, and Key Research and Development Project of Zhejiang Province in China under Grant 2021C03137, Zhejiang Provincial Natural Science Foundation of China under Grant LY21F020001, Science and Technology Plan Project of Wenzhou in China under Grant ZG2020026.

Funding

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and Affiliations

  1. Department of Digital Media Technology, Hangzhou Dianzi University, Hangzhou, 310018, China

    Guangyi Zhao, Simin Kou, Xuesong Yin, Guodao Zhang & Yigang Wang

Authors
  1. Guangyi Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simin Kou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuesong Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guodao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yigang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xuesong Yin.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhao, G., Kou, S., Yin, X. et al. Self-supervised deep subspace clustering with entropy-norm. Cluster Comput 27, 1611–1623 (2024). https://doi.org/10.1007/s10586-023-04033-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-023-04033-7

Keywords

Search

Navigation