Skip to main content
SpringerLink
Log in

LGAT: a light graph attention network focusing on message passing for semi-supervised node classification

  • Regular Paper
  • Published:
Computing Aims and scope Submit manuscript

Abstract

Deep learning has shown superior performance in various applications. The emergence of graph convolution neural networks (GCNs) enables deep learning to learn the latent representation from graph-structured data with rich attributes. To be specific, the message passing mechanism of GCNs can aggregate and update messages through the topological relationship between nodes in a graph. The graph attention network (GAT) introduces the attention mechanism into GCNs when aggregating messages and achieves significant performance on the node classification task. However, focusing on each node in the neighborhood, GAT becomes extremely complex. In addition, although stacking network layers could obtain a wider receptive field, it also brings high time cost and leads to the difficulty of training. To handle this problem, this paper only divides the messages into two types, i.e. self message and neighborhood message. The neighborhood message comes from the neighborhood with(out) self-loop while the self message comes from the node itself. Then, we design a light attention mechanism that only focuses on two weights, one for the self message, and the other for the neighborhood message, to adaptively reveal the different contributions of messages from a node as well as its neighborhood. In addition, we also adopt linear propagation, a shallow and efficient method, to aggregate messages from distant neighbors and thus obtain a wider neighborhood receiving field. To verify the effectiveness of our proposed approach, extensive experiments have been conducted on the semi-supervised node classification task. Results show that our proposed approach achieves comparable or even better performance than the baseline methods with complicated GCN structures on the benchmark datasets. Specifically, the proposed light attention mechanism focusing on message passing exhibits a great efficiency improvement with the training time cost less than half of GAT.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Cao S, Lu W, Xu Q (2016) Deep neural networks for learning graph representations. In: proceedings of the AAAI conference on artificial intelligence, vol 30

  2. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105

    Google Scholar 

  3. Cho K, Van Merriënboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder–decoder for statistical machine translation. arXiv preprint arXiv:1406.1078

  4. Wu Z, Pan S, Chen F, Long G, Zhang C, Philip SY (2020) A comprehensive survey on graph neural networks. IEEE Trans Neural Netw Learn Syst 32(1):4–24

    Article  MathSciNet  Google Scholar 

  5. Zhou J, Cui G, Hu S, Zhang Z, Yang C, Liu Z, Wang L, Li C, Sun M (2020) Graph neural networks: a review of methods and applications. AI Open 1:57–81

    Article  Google Scholar 

  6. Cai H, Zheng VW, Chang KC-C (2018) A comprehensive survey of graph embedding: problems, techniques, and applications. IEEE Trans Knowl Data Eng 30(9):1616–1637

    Article  Google Scholar 

  7. Zhang Z, Cui P, Zhu W (2020) Deep learning on graphs: a survey. IEEE Trans Knowl Data Eng 34(1):249–270

    Article  Google Scholar 

  8. Zhang D, Yin J, Zhu X, Zhang C (2018) Network representation learning: a survey. IEEE Trans Big Data 6(1):3–28

    Article  Google Scholar 

  9. Roweis ST, Saul LK (2000) Nonlinear dimensionality reduction by locally linear embedding. Science 290(5500):2323–2326

    Article  CAS  PubMed  ADS  Google Scholar 

  10. Belkin M, Niyogi P (2001) Laplacian eigenmaps and spectral techniques for embedding and clustering. Adv Neural Inf Process Syst 14:585–591

    Google Scholar 

  11. Grover A, Leskovec J ( 2016) node2vec: Scalable feature learning for networks. In: proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp 855– 864

  12. Perozzi B, Al-Rfou R, Skiena S ( 2014) Deepwalk: online learning of social representations. In: proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining, pp 701– 710

  13. Tang J, Qu M, Wang M, Zhang M, Yan J, Mei Q ( 2015) Line: large-scale information network embedding. In: proceedings of the 24th international conference on World Wide Web, pp 1067– 1077

  14. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

  15. Hamilton WL, Ying R, Leskovec J (2017) Inductive representation learning on large graphs. In: proceedings of the 31st international conference on neural information processing systems, pp 1025– 1035

  16. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907

  17. Veličković P, Cucurull G, Casanova A, Romero A, Lio P, Bengio Y (2017) Graph attention networks. arXiv preprint arXiv:1710.10903

  18. Chen Z, Chen F, Zhang L, Ji T, Fu K, Zhao L, Chen F, Wu L, Aggarwal C, Lu C-T (2020) Bridging the gap between spatial and spectral domains: a survey on graph neural networks. arXiv preprint arXiv:2002.11867

  19. Bruna J, Zaremba W, Szlam A, LeCun Y (2013) Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203

  20. Defferrard M, Bresson X, Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering. Adv Neural Inf Process Syst 29:3844–3852

    Google Scholar 

  21. Gilmer J, Schoenholz SS, Riley PF, Vinyals O, Dahl GE ( 2017) Neural message passing for quantum chemistry. In: international conference on machine learning, PMLR, pp 1263– 1272

  22. Xu K, Hu W, Leskovec J, Jegelka S (2018) How powerful are graph neural networks?. arXiv preprint arXiv:1810.00826

  23. Wu F, Souza A, Zhang T, Fifty C, Yu T, Weinberger K ( 2019) Simplifying graph convolutional networks. In: international conference on machine learning, PMLR, pp 6861– 6871

  24. Zhu Z, Peng J, Li J, Chen L, Yu Q, Luo S (2022) Spiking graph convolutional networks. arXiv preprint arXiv:2205.02767

  25. Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G (2008) The graph neural network model. IEEE Trans Neural Netw 20(1):61–80

    Article  PubMed  Google Scholar 

  26. Ruiz L, Gama F, Ribeiro A (2020) Gated graph recurrent neural networks. IEEE Trans Signal Process 68:6303–6318

    Article  MathSciNet  ADS  Google Scholar 

  27. Guo M-H, Xu T-X, Liu J-J, Liu Z-N, Jiang P-T, Mu T-J, Zhang S-H, Martin RR, Cheng M-M, Hu S-M (2022) Attention mechanisms in computer vision: a survey. Comput Vis Media 8:1–38

    Google Scholar 

  28. Zhang J, Shi X, Xie J, Ma H, King I, Yeung D-Y (2018) Gaan: gated attention networks for learning on large and spatiotemporal graphs. arXiv preprint arXiv:1803.07294

  29. Wang X, Ji H, Shi C, Wang B, Ye Y, Cui P, Yu PS (2019) Heterogeneous graph attention network. In: The World Wide Web conference, pp 2022– 2032

  30. Wang B, Jia J, Gong NZ (2021) Semi-supervised node classification on graphs: Markov random fields vs. graph neural networks. In: proceedings of the AAAI conference on artificial intelligence, vol 35. pp 10093–10101

  31. Cramer JS (2002) The origins of logistic regression

  32. Pal SK, Mitra S (1992) Multilayer perceptron, fuzzy sets, classifiaction

  33. Mernyei P, Cangea C(2020) Wiki-cs: a wikipedia-based benchmark for graph neural networks. CoRR arXiv: abs/2007.02901

  34. Shchur O, Mumme M, Bojchevski A, Günnemann S (2018) Pitfalls of graph neural network evaluation. CoRR arXiv: abs/1811.05868

  35. Van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(86):2579–2605

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No.61876186, No.61977061) and the Project of Xuzhou Science and Technology (No.KC21300).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, China University of Mining and Technology, Xuzhou, 221116, Jiangsu, China

    Chengcheng Sun, Fanrong Meng, Chenhao Li, Xiaobin Rui & Zhixiao Wang

  2. Mine Digitization Engineering Research Center of the Ministry of Education, Xuzhou, 221116, Jiangsu, China

    Fanrong Meng, Xiaobin Rui & Zhixiao Wang

Authors
  1. Chengcheng Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fanrong Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chenhao Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiaobin Rui
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhixiao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhixiao Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, C., Meng, F., Li, C. et al. LGAT: a light graph attention network focusing on message passing for semi-supervised node classification. Computing (2024). https://doi.org/10.1007/s00607-024-01261-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00607-024-01261-6

Keywords

Mathematics Subject Classification

Search

Navigation