Skip to main content
SpringerLink
Log in

Self-supervised graph clustering via attention auto-encoder with distribution specificity

  • Regular Paper
  • Published:
Multimedia Systems Aims and scope Submit manuscript

Abstract

Graph clustering, an essential unsupervised learning task in data mining, has garnered significant attention in recent years. With the advent of deep learning, considerable progress has been made in this field. However, existing methods present several limitations: (1) Most encoder models employ Graph Convolutional Networks (GCNs) as encoders. However, GCNs assign equal weight to each neighboring node and have been shown to be oversmoothing, thereby impacting clustering performance. (2) Most algorithms do not fully utilize the original graph content and structural information, leading to incomplete embedding features. (3) These methods do not account for the specific distribution of clustering of embedding features and the enhancement of staged pseudo-labels on clustering tasks.In this study,we propose a novel end-to-end graph clustering model that leverages graph attention encoders. Specifically, we initially employ a graph attention encoder to extract the inherent information from the original graph. This process assigns varying weights to different nodes, thereby avoiding excessive smoothing. We also fully utilize the guidance of periodic pseudo-labels to facilitate the learning of potential features that are beneficial for clustering. In addition, to improve the model’s clustering performance, we introduce a regularization term that distributes the node features of different classifications across distinct low-dimensional spaces. Furthermore, to prevent the embedding features from straying from the original graph features, we design an information consistency module. Experimental results on the node graph datasets show that our model outperforms other state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

The data sets that support the findings of this study are available from the author upon request. Declaration of interests

No datasets were generated or analysed during the current study.

References

  1. Min, E., Guo, X., Liu, Q., Zhang, G., Cui, J., Long, J.: A survey of clustering with deep learning: from the perspective of network architecture. IEEE Access 6, 39501–39514 (2018)

    Article  Google Scholar 

  2. Xia, F., Sun, K., Yu, S., Aziz, A., Wan, L., Pan, S., Liu, H.: Graph learning: a survey. IEEE Trans. Artif. Intell. 2(2), 109–127 (2021)

    Article  Google Scholar 

  3. Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk: Online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 701–710 (2014)

  4. Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., Mei, Q.: Line: large-scale information network embedding. In: Proceedings of the 24th International Conference on World Wide Web, pp. 1067–1077 (2015)

  5. Zhu, W., Wang, X., Cui, P.: In: Pedrycz, W., Chen, S.-M. (eds.) Deep learning for learning graph representations, pp. 169– 210. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-31756-06

  6. Wu, S., Tang, Y., Zhu, Y., Wang, L., Xie, X., Tan, T.: Session-based recommendation with graph neural networks. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 346–353 (2019)

  7. Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., Eliassi-Rad, T.: Collective classification in network data. AI Mag. 29(3), 93–93 (2008)

    Google Scholar 

  8. Ju, W., Yi, S., Wang, Y., Long, Q., Luo, J., Xiao, Z., Zhang, M.: A survey of data-efficient graph learning. (2024). https://doi.org/10.48550/arXiv.2402.00447

  9. Parkhi, O., Vedaldi, A., Zisserman, A.: Deep face recognition. In: BMVC 2015-Proceedings of the British Machine Vision Conference 2015 (2015). British Machine Vision Association

  10. Zhang, S., Tong, H., Xu, J., Maciejewski, R.: Graph convolutional networks: a comprehensive review. Comput. Soc. Netw. 6(1), 1–23 (2019)

    Article  Google Scholar 

  11. Ju, W., Fang, Z., Gu, Y., Liu, Z., Long, Q., Qiao, Z., Qin, Y., Shen, J., Sun, F., Xiao, Z., Yang, J., Yuan, J., Zhao, Y., Wang, Y., Luo, X., Zhang, M.: A comprehensive survey on deep graph representation learning. Neural Networks 173, (2024). https://doi.org/10.1016/j.neunet.2024.106207

    Article  Google Scholar 

  12. Kipf, T.N., Welling, M.: Variational graph auto-encoders. stat 1050, 21 (2016)

    Google Scholar 

  13. Pan, S., Hu, R., Long, G., Jiang, J., Yao, L., Zhang, C.: Adversarially regularized graph autoencoder for graph embedding. In: International Joint Conference on Artificial Intelligence 2018, pp. 2609–2615 (2018). Association for the Advancement of Artificial Intelligence (AAAI)

  14. Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. stat 1050, 4 (2018)

    Google Scholar 

  15. Wang, C., Pan, S., Hu, R., Long, G., Jiang, J., Zhang, C.: Attributed graph clustering: a deep attentional embedding approach. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3670–3676 (2019)

  16. Vaswani, A., Shazeer, N., Parmar, N., Uszko reit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. NIPS’17, pp. 6000–6010. Curran Associates Inc., Red Hook, NY, USA (2017).

  17. Cai, H., Zheng, V.W., Chang, K.C.-C.: A comprehensive survey of graph embedding: problems, techniques, and applications. IEEE Trans. Knowl. Data Eng. 30(9), 1616–1637 (2018)

    Article  Google Scholar 

  18. Grady, L.: Random walks for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 28(11), 1768–1783 (2006)

    Article  Google Scholar 

  19. Grover, A., Leskovec, J.: node2vec: scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855–864 (2016)

  20. Cao, S., Lu, W., Xu, Q.: Grarep: learning graph representations with global structural information. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 891–900 (2015)

  21. Ou, M., Cui, P., Pei, J., Zhang, Z., Zhu, W.: Asymmetric transitivity preserving graph embedding. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1105–1114 (2016)

  22. Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Trans. Neural Netw. 20(1), 61–80 (2008)

    Article  Google Scholar 

  23. Xia, W., Wang, Q., Gao, Q., Zhang, X., Gao, X.: Self-supervised graph convolutional network for multi-view clustering. IEEE Trans. Multimed. 24, 3182–3192 (2022). https://doi.org/10.1109/TMM.2021.3094296

    Article  Google Scholar 

  24. Cheng, J., Wang, Q., Tao, Z., Xie, D., Gao, Q.: Multi-view attribute graph convolution networks for clustering. In: Proceedings of the Twenty-ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 2973–2979 (2021)

  25. Sun, D., Liu, L., Luo, B., Ding, Z.: Glass: a graph laplacian autoencoder with subspace clustering regularization for graph clustering. Cogn. Comput. 15(3), 803–821 (2023)

    Article  Google Scholar 

  26. Ju, W., Gu, Y., Chen, B., Sun, G., Qin, Y., Liu, X., Luo, X., Zhang, M.: Glcc: A general framework for graph-level clustering. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 4391–4399 (2023)

  27. Luo, X., Ju, W., Gu, Y., Mao, Z., Liu, L., Yuan, Y., Zhang, M.: Self-supervised graph-level representation learning with adversarial contrastive learning. ACM Trans. Knowl. Discov. Data 18(2), 1–23 (2023)

    Article  Google Scholar 

  28. Zhang, H., Li, P., Zhang, R., Li, X.: Embedding graph auto-encoder for graph clustering. IEEE Trans. Neural Netw. Learn. Syst. 34(11), 9352–9362 (2023). https://doi.org/10.1109/TNNLS.2022.3158654

    Article  MathSciNet  Google Scholar 

  29. Salehi, A., Davulcu, H.: Graph attention auto-encoders. In: 32nd IEEE International Conference on Tools with Artificial Intelligence, ICTAI 2020, pp. 989–996 (2020). IEEE Computer Society

  30. Yi, S., Ju, W., Qin, Y., Luo, X., Liu, L., Zhou, Y., Zhang, M.: Redundancy-free self-supervised relational learning for graph clustering. IEEE Transactions on Neural Networks and Learning Systems (2023)

  31. Tian, F., Gao, B., Cui, Q., Chen, E., Liu, T.-Y.: Learning deep representations for graph clustering. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 28 (2014)

  32. Xie, J., Girshick, R., Farhadi, A.: Unsupervised deep embedding for clustering analysis. In: International Conference on Machine Learning, pp. 478–487 (2016). PMLR

  33. Park, J., Lee, M., Chang, H.J., Lee, K., Choi, J.Y.: Symmetric graph convolutional autoencoder for unsupervised graph representation learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6519–6528 (2019)

  34. Tu, W., Zhou, S., Liu, X., Guo, X., Cai, Z., Zhu, E., Cheng, J.: Deep fusion clustering network. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 9978–9987 (2021)

  35. Bo, D., Wang, X., Shi, C., Zhu, M., Lu, E., Cui, P.: Structural deep clustering network. In: Proceedings of the Web Conference 2020, pp. 1400–1410 (2020)

  36. Dubey, S.R., Singh, S.K., Chaudhuri, B.B.: Activation functions in deep learning: a comprehensive survey and benchmark. Neurocomputing 503, 92–108 (2022)

    Article  Google Scholar 

  37. Krishna, K., Murty, M.N.: Genetic k-means algorithm. IEEE Trans. Syst. Man Cybern. Part B (Cybernetics) 29(3), 433–439 (1999)

    Article  Google Scholar 

  38. Von Luxburg, U.: A tutorial on spectral clustering. Stat. Comput. 17, 395–416 (2007)

    Article  MathSciNet  Google Scholar 

  39. Yang, C., Liu, Z., Zhao, D., Sun, M., Chang, E.Y.: Network representation learning with rich text information. In: Proceedings of the 24th International Conference on Artificial Intelligence. IJCAI’15, pp. 2111–2117. AAAI Press, Buenos Aires (2015)

  40. Wang, C., Pan, S., Long, G., Zhu, X., Jiang, J.: Mgae: marginalized graph autoencoder for graph clustering. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 889–898 (2017)

  41. Cui, G., Zhou, J., Yang, C., Liu, Z.: Adaptive graph encoder for attributed graph embedding. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 976–985 (2020)

  42. Sun, D., Li, D., Ding, Z., Zhang, X., Tang, J.: Dual-decoder graph autoencoder for unsupervised graph representation learning. Knowl.-Based Syst. 234, 107564 (2021)

    Article  Google Scholar 

  43. Xie, X., Chen, W., Kang, Z., Peng, C.: Contrastive graph clustering with adaptive filter. Expert Syst. Appl. 219, 119645 (2023)

    Article  Google Scholar 

  44. Kou, S., Xia, W., Zhang, X., Gao, Q., Gao, X.: Self-supervised graph convolutional clustering by preserving latent distribution. Neurocomputing 437, 218–226 (2021)

    Article  Google Scholar 

  45. Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of machine learning research 9(11), (2008)

Download references

Acknowledgements

This work is sponsored by National Natural Science Foundation of China (CN) under grant numbers 62276164 and 61602296, ‘Chenguang Program’ supported by Shanghai Education Development Foundation and Shanghai Municipal Education Commission under grant number 18CG54. Furthermore, this work is also supported by ‘Science and technology innovation action plan’ Natural Science Foundation of Shanghai under grant number 22ZR1427000. The authors would like to thank their supports.

Author information

Authors and Affiliations

  1. College of Information Engineering, Shanghai Maritime University, Shanghai, 201306, China

    Zishi Li & Changming Zhu

Authors
  1. Zishi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Changming Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Zishi Li is responsible for the whole process of paper implementation, including but not limited to code writing, experimental process, data summary, and paper writing. Changming Zhuis responsible for code guidance and paper review?

Corresponding author

Correspondence to Changming Zhu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict ofinterest.

Additional information

Communicated by B. Bao.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Z., Zhu, C. Self-supervised graph clustering via attention auto-encoder with distribution specificity. Multimedia Systems 30, 150 (2024). https://doi.org/10.1007/s00530-024-01346-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00530-024-01346-4

Keywords

Search

Navigation