Skip to main content
SpringerLink
Log in

Scalable decoupling graph neural network with feature-oriented optimization

  • Regular Paper
  • Published:
The VLDB Journal Aims and scope Submit manuscript

Abstract

Recent advances in data processing have stimulated the demand for learning graphs of very large scales. Graph neural networks (GNNs), being an emerging and powerful approach in solving graph learning tasks, are known to be difficult to scale up. Most scalable models apply node-based techniques in simplifying the expensive graph message-passing propagation procedure of GNNs. However, we find such acceleration insufficient when applied to million- or even billion-scale graphs. In this work, we propose SCARA, a scalable GNN with feature-oriented optimization for graph computation. SCARA efficiently computes graph embedding from the dimension of node features, and further selects and reuses feature computation results to reduce overhead. Theoretical analysis indicates that our model achieves sub-linear time complexity with a guaranteed precision in propagation process as well as GNN training and inference. We conduct extensive experiments on various datasets to evaluate the efficacy and efficiency of SCARA. Performance comparison with baselines shows that SCARA can reach up to \(800\times \) graph propagation acceleration than current state-of-the-art methods with fast convergence and comparable accuracy. Most notably, it is efficient to process precomputation on the largest available billion-scale GNN dataset Papers100M (111 M nodes, 1.6 B edges) in 13 s.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Fig. 2
Algorithm 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. The source code and data used in the paper have been made available at: https://github.com/gdmnl/SCARA-PPR

References

  1. Al-Rfou, R., Perozzi, B., Zelle, D.: Ddgk: learning graph representations for deep divergence graph kernels. In: The World Wide Web Conference, pp. 37–48 (2019)

  2. Andersen, R., Borgs, C., Chayes, J., Hopcraft, J., Mirrokni, V.S., Teng, S.H.: Local computation of pagerank contributions. In: Algorithms and models for the web-graph, vol. 4863, pp. 150–165. Springer Berlin Heidelberg, Berlin, Heidelberg (2007)

  3. Andersen, R., Chung, F., Lang, K.: Local graph partitioning using PageRank vectors. In: 2006 47th Annual IEEE Symposium on Foundations of Computer Science (FOCS’06), pp. 475–486. IEEE (2006)

  4. Atwood, J., Towsley, D.: Diffusion-convolutional neural networks. In: 29th Advances in Neural Information Processing Systems pp. 2001–2009 (2016)

  5. Bojchevski, A., Klicpera, J., Perozzi, B., Kapoor, A., Blais, M., Rózemberczki, B., Lukasik, M., Günnemann, S.: Scaling graph neural networks with approximate pagerank. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 2464–2473 (2020)

  6. Candès, E.J., Li, X., Ma, Y., Wright, J.: Robust principal component analysis? J. ACM 58(3), 1–37 (2011)

    Article  MathSciNet  Google Scholar 

  7. Chandrasekaran, V., Sanghavi, S., Parrilo, P.A., Willsky, A.S.: Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim. 21(2), 572–596 (2011)

    Article  MathSciNet  Google Scholar 

  8. Chen, Z., Bruna, J., Li, L.: Supervised community detection with line graph neural networks. In: 7th International Conference on Learning Representations (2019)

  9. Chen, J., Ma, T., Xiao, C.: Fastgcn: fast learning with graph convolutional networks via importance sampling. In: International Conference on Learning Representations (2018)

  10. Chen, M., Wei, Z., Ding, B., Li, Y., Yuan, Y., Du, X., Wen, J.R.: Scalable graph neural networks via bidirectional propagation. In: 33rd Advances in Neural Information Processing Systems (2020)

  11. Chen, J., Zhu, J., Song, L.: Stochastic training of graph convolutional networks with variance reduction. Int. C. Mach. Learn. 3, 1503–1532 (2018)

    Google Scholar 

  12. Chiang, W.L., Liu, X., Si, S., Li, Y., Bengio, S., Hsieh, C.J.: Cluster-gcn: An efficient algorithm for training deep and large graph convolutional networks. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 257–266 (2019)

  13. Chien, E., Peng, J., Li, P., Milenkovic, O.: Adaptive universal generalized pagerank graph neural network. In: 9th International Conference on Learning Representations (2021)

  14. Coudron, M., Lerman, G.: On the sample complexity of robust PCA. In: 25th Advances in Neural Information Processing Systems (2012)

  15. Ding, M., Kong, K., Li, J., Zhu, C., Dickerson, J.P., Huang, F., Goldstein, T.: VQ-GNN: a universal framework to scale up graph neural networks using vector quantization. In: 34th Advances in Neural Information Processing Systems (2021)

  16. Fey, M., Lenssen, J.E., Weichert, F., Leskovec, J.: GNNAutoScale: scalable and expressive graph neural networks via historical embeddings. In: 38th International Conference on Machine Learning. PMLR 139 (2021)

  17. Fogaras, D., Rácz, B., Csalogány, K., Sarlós, T.: Towards scaling fully personalized pagerank: algorithms, lower bounds, and experiments. Inter. Math. 2(3), 333–358 (2005)

    MathSciNet  Google Scholar 

  18. Hamilton, W.L., Ying, R., Leskovec, J.: Inductive representation learning on large graphs. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 1025–1035 (2017)

  19. Hu, W., Fey, M., Zitnik, M., Dong, Y., Ren, H., Liu, B., Catasta, M., Leskovec, J., Barzilay, R., Battaglia, P., Bengio, Y., Bronstein, M., Günnemann, S., Hamilton, W., Jaakkola, T., Jegelka, S., Nickel, M., Re, C., Song, L., Tang, J., Welling, M., Zemel, R.: Open graph benchmark : datasets for machine learning on graphs. In: 33rd Advances in Neural Information Processing Systems (2020)

  20. Huang, Z., Zhang, S., Xi, C., Liu, T., Zhou, M.: Scaling up graph neural networks via graph coarsening. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, vol. 1, pp. 675–684 (2021)

  21. Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations (2017)

  22. Klicpera, J., Bojchevski, A., Günnemann, S.: Predict then propagate: graph neural networks meet personalized pagerank. In: 7th International Conference on Learning Representations, pp. 1–15 (2019)

  23. Li, X., Zhu, R., Cheng, Y., Shan, C., Luo, S., Li, D., Qian, W.: Finding global homophily in graph neural networks when meeting heterophily. In: 39th International Conference on Machine Learning (2022)

  24. Liao, N., Luo, S., Li, X., Shi, J.: LD2: Scalable heterophilous graph neural network with decoupled embedding. In: Advances in Neural Information Processing Systems, vol. 36 (2023)

  25. Liao, N., Mo, D., Luo, S., Li, X., Yin, P.: SCARA: scalable graph neural networks with feature-oriented optimization. Proc. VLDB Endowm. 15(11), 3240–3248 (2022)

    Article  Google Scholar 

  26. Lin, Z., Liu, R., Su, Z.: Linearized alternating direction method with adaptive penalty for low-rank representation. In: 24th Advances in Neural Information Processing Systems (2011)

  27. Lin, D., Wong, R.C.W., Xie, M., Wei, V.J.: Index-free approach with theoretical guarantee for efficient random walk with restart query. In: 2020 IEEE 36th International Conference on Data Engineering (ICDE), pp. 913–924 (2020)

  28. Liu, H., Liao, N., Luo, S.: Simga: A simple and effective heterophilous graph neural network with efficient global aggregation. arXiv e-prints (2023)

  29. Mo, D., Luo, S.: Agenda: robust personalized pageranks in evolving graphs. In: Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 1315–1324. ACM, Virtual Event Queensland Australia (2021)

  30. Page, L., Brin, S., Motwani, R., Winograd, T.: The pagerank citation ranking: bringing order to the web. Tech. rep. (1999)

  31. Sinha, A., Shen, Z., Song, Y., Ma, H., Eide, D., Hsu, B.J.P., Wang, K.: An overview of microsoft academic service (mas) and applications. In: Proceedings of the 24th International Conference on World Wide Web, pp. 243–246 (2015)

  32. Sun, L., Dou, Y., Yang, C., Wang, J., Yu, P.S., He, L., Li, B.: Adversarial attack and defense on graph data: a survey. arXiv e-prints (2018)

  33. Thekumparampil, K.K., Wang, C., Oh, S., Li, L.J.: Attention-based graph neural network for semi-supervised learning. arXiv e-prints (2018)

  34. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. In: 8th International Conference on Learning Representations (2017)

  35. Wang, H., He, M., Wei, Z., Wang, S., Yuan, Y., Du, X., Wen, J.R.: Approximate graph propagation. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, vol. 1, pp. 1686–1696. Association for Computing Machinery (2021)

  36. Wang, K., Luo, S., Lin, D.: River of no return: graph percolation embeddings for efficient knowledge graph reasoning. arXiv e-prints (2023)

  37. Wang, C., Pan, S., Long, G., Zhu, X., Jiang, J.: Mgae: marginalized graph autoencoder for graph clustering. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, CIKM ’17, p. 889–898. Association for Computing Machinery, New York, NY, USA (2017)

  38. Wang, X., Zhang, M.: How powerful are spectral graph neural networks. In: 39th International Conference on Machine Learning (2022)

  39. Wang, S., Yang, R., Xiao, X., Wei, Z., Yang, Y.: Fora: simple and effective approximate single-source personalized pagerank. Proc. ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining Part F1296, 505–514 (2017)

    Google Scholar 

  40. Wang, S., Yang, R., Wang, R., Xiao, X., Wei, Z., Lin, W., Yang, Y., Tang, N.: Efficient algorithms for approximate single-source personalized PageRank queries. ACM Trans. Datab. Syst. 44(4), 1–37 (2019)

    Article  MathSciNet  Google Scholar 

  41. Wu, H., Gan, J., Wei, Z., Zhang, R.: Unifying the global and local approaches: an efficient power iteration with forward push. In: Proceedings of the 2021 International Conference on Management of Data, vol. 1, pp. 1996–2008 (2021)

  42. Wu, F., Souza, A., Zhang, T., Fifty, C., Yu, T., Weinberger, K.: Simplifying graph convolutional networks. In: K. Chaudhuri, R. Salakhutdinov (eds.) Proceedings of the 36th International Conference on Machine Learning, vol. 97, pp. 6861–6871 (2019)

  43. Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., Yu, P.S.: A comprehensive survey on graph neural networks. IEEE Trans. Neur. Netw. Learn. Syst. 32(1), 4–24 (2021)

    Article  MathSciNet  Google Scholar 

  44. Yang, R., Shi, J., Xiao, X., Yang, Y., Liu, J., Bhowmick, S.S.: Scaling attributed network embedding to massive graphs. Proc. VLDB Endow. 14(1), 37–49 (2021)

    Article  Google Scholar 

  45. Ying, R., He, R., Chen, K., Eksombatchai, P., Hamilton, W.L., Leskovec, J.: Graph convolutional neural networks for web-scale recommender systems. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 974–983 (2018)

  46. Zeng, H., Zhou, H., Srivastava, A., Kannan, R., Prasanna, V.: Graphsaint: Graph sampling based learning method. In: International Conference on Learning Representations (2019)

  47. Zhang, J., Zhang, H., Xia, C., Sun, L.: Graph-bert: only attention is needed for learning graph representations. arXiv e-prints (2020)

  48. Zhang, M., Chen, Y.: Link prediction based on graph neural networks. Adv. Neural Inf. Process. Syst. 31, 5165–5175 (2018)

  49. Zhang, Z., Cui, P., Zhu, W.: Deep learning on graphs: a survey. IEEE Tran. Knowl. Data Eng. 14(8), 1 (2020)

    Google Scholar 

  50. Zhu, Z., Peng, J., Li, J., Chen, L., Yu, Q., Luo, S.: Spiking graph convolutional networks. In: 31th International Joint Conference on Artificial Intelligence (2022)

  51. Zügner, D., Akbarnejad, A., Günnemann, S.: Adversarial attacks on neural networks for graph data. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2847–2856 (2018)

Download references

Acknowledgements

This research is supported by Singapore MOE funding (MOE-T2EP20122-0003, RS05/21), NTU-NAP Startup Grant (022029-00001), and the Joint NTU-WeBank Research Centre on FinTech. Xiang Li is supported by Shanghai Science and Technology Committee General Program No. 22ZR1419900 and National Natural Science Foundation of China No. 62202172.

Author information

Authors and Affiliations

  1. Nanyang Technological University, Singapore, Singapore

    Ningyi Liao, Dingheng Mo & Siqiang Luo

  2. East China Normal University, Shanghai, China

    Xiang Li

  3. Google Research, San Francisco, USA

    Pengcheng Yin

Authors
  1. Ningyi Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dingheng Mo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Siqiang Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Pengcheng Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Siqiang Luo.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liao, N., Mo, D., Luo, S. et al. Scalable decoupling graph neural network with feature-oriented optimization. The VLDB Journal 33, 667–683 (2024). https://doi.org/10.1007/s00778-023-00829-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00778-023-00829-6

Keywords

Search

Navigation