Skip to main content
SpringerLink
Log in

Binarized graph neural network

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Recently, there have been some breakthroughs in graph analysis by applying the graph neural networks (GNNs) following a neighborhood aggregation scheme, which demonstrate outstanding performance in many tasks. However, we observe that the parameters of the network and the embedding of nodes are represented in real-valued matrices in existing GNN-based graph embedding approaches which may limit the efficiency and scalability of these models. It is well-known that binary vector is usually much more space and time efficient than the real-valued vector. This motivates us to develop a binarized graph neural network to learn the binary representations of the nodes with binary network parameters following the GNN-based paradigm. Our proposed method can be seamlessly integrated into the existing GNN-based embedding approaches to binarize the model parameters and learn the compact embedding. Extensive experiments indicate that the proposed binarized graph neural network, namely BGN, is orders of magnitude more efficient in terms of both time and space while matching the state-of-the-art performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. http://snap.stanford.edu/proj/embeddings-www

References

  1. Bai, Y., Ding, H., Bian, S., Chen, T., Sun, Y., Wang, W.: Simgnn: a neural network approach to fast graph similarity computation. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, VIC, Australia, February 11-15, 2019, pp. 384–392 (2019)

  2. Bengio, Y., Léonard, N., Courville, A.: Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432 (2013)

  3. Chen, G., He, S., Meng, H., Huang, K.: Phonebit: Efficient Gpu-Accelerated binary neural network inference engine for mobile phones. In: 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 786–791. IEEE (2020)

  4. Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., Bengio, Y.: Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. Neural Information Processing Systems neurIPS (2016)

  5. Cui, P., Wang, X., Pei, J., Zhu, W.: A survey on network embedding. IEEE Trans. Knowl. Data Eng. (TKDE) 31(5), 833–852 (2019)

    Article  Google Scholar 

  6. Erdös, P., Rényi, A.: On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci 5(1), 17–60 (1960)

    MathSciNet  MATH  Google Scholar 

  7. Fan, R., Chang, K., Hsieh, C., Wang, X., Lin, C.: LIBLINEAR: A library for large linear classification. JMLR 9, 1871–1874 (2008)

    MATH  Google Scholar 

  8. Gong, R., Liu, X., Jiang, S., Li, T., Hu, P., Lin, J., Yu, F., Yan, J.: Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 4852–4861 (2019)

  9. Grover, A., Leskovec, J.: Node2vec: Scalable feature learning for networks. In: ACM SIGKDD, pp. 855–864 (2016)

  10. Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: Neural Information Processing Systems NeurIPS, pp. 1024–1034 (2017)

  11. Hamilton, W.L., Ying, R., Leskovec, J.: Representation learning on graphs: Methods and applications. IEEE Data Eng. Bull. 40(3), 52–74 (2017)

    Google Scholar 

  12. Huang, W., Zhang, T., Rong, Y., Huang, J.: Adaptive sampling towards fast graph representation learning. In: Neural Information Processing Systems NeurIPS, pp. 4563–4572 (2018)

  13. Hubara, I., Courbariaux, M., Soudry, D., El-yaniv, R. , Bengio, Y.: Binarized neural networks. In: Neural Information Processing Systems NeurIPS, pp. 4107–4115 (2016)

  14. Kazemi, S.M., Poole, D.: Simple embedding for link prediction in knowledge graphs. In: Neural Information Processing Systems NeurIPS, pp. 4289–4300 (2018)

  15. Kipf, T.N., Welling, M.: Semi-Supervised classification with graph convolutional networks. In: ICLR (2017)

  16. Leskovec, J., Krevl, A.: SNAP Datasets: Stanford large network dataset collection http://snap.stanford.edu/data (2014)

  17. Li, Y., Gong, R., Yu, F., Dong, X., Liu, X.: Dms: Differentiable dimension search for binary neural networks

  18. Li, Y., Gu, C., Dullien, T., Vinyals, O., Kohli, P.: Graph matching networks for learning the similarity of graph structured objects. In: Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 3835–3845 (2019)

  19. Li, Y., Tarlow, D., Brockschmidt, M., Zemel, R.: Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493 (2015)

  20. Lian, D., Zheng, K., Zheng, V.W., Ge, Y., Cao, L., Tsang, I.W., Xie, X.: High-Order proximity preserving information network hashing. In: ACM SIGKDD, pp. 1744–1753 (2018)

  21. Liu, H., Wang, R., Shan, S., Chen, X.: Deep supervised hashing for fast image retrieval. In: CVPR, pp. 2064–2072 (2016)

  22. Liu, W., Mu, C., Kumar, S., Chang, S.: Discrete graph hashing. In: Neural Information Processing Systems NeurIPS, pp. 3419–3427 (2014)

  23. Liu, Z., Shen, Z., Savvides, M., Cheng, K.: Reactnet: Towards precise binary neural network with generalized activation functions. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J. (eds.) Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV, Lecture Notes in Computer Science, vol. 12359, pp 143–159. Springer (2020)

  24. Martínez, B., Yang, J., Bulat, A., Tzimiropoulos, G.: Training binary neural networks with real-to-binary convolutions. In: 8Th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. Openreview.Net (2020)

  25. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Neural Information Processing Systems NeurIPS, pp. 3111–3119 (2013)

  26. Perozzi, B., Al-Rfou, R., Skiena, S.: Deepwalk: Online learning of social representations. In: ACM SIGKDD, pp. 701–710. ACM (2014)

  27. Qin, H., Gong, R., Liu, X., Bai, X., Song, J., Sebe, N.: Binary neural networks: A survey. Pattern Recognition, pp. 107281 (2020)

  28. Qin, H., Gong, R., Liu, X., Shen, M., Wei, Z., Yu, F., Song, J.: Forward and backward information retention for accurate binary neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR, pp. 2250–2259 (2020)

  29. Qin, J., Wang, Y., Xiao, C., Wang, W., Lin, X., Ishikawa, Y.: GPH: Similarity search in hamming space. In: IEEE ICDE, pp. 29–40 (2018)

  30. Qiu, J., Dong, Y., Ma, H., Li, J., Wang, K., Tang, J.: Network embedding as matrix factorization: Unifying deepwalk, Line, Pte, and Node2vec. In: ACM WSDM, pp. 459–467 (2018)

  31. Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: Xnor-Net: Imagenet classification using binary convolutional neural networks. In: European Conference on Computer Vision, pp. 525–542. Springer (2016)

  32. Ribeiro, L.F., Saverese, P.H., Figueiredo, D.R.: Struc2vec: Learning node representations from structural identity. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 385–394 (2017)

  33. Salakhutdinov, R., Hinton, G.E.: Semantic hashing. Int. J. Approx. Reasoning 50(7), 969–978 (2009)

    Article  Google Scholar 

  34. Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., Eliassi-Rad, T.: Collective classification in network data. AI Magazine 29(3), 93–93 (2008)

    Article  Google Scholar 

  35. Shen, F., Shen, C., Liu, W., Shen, H.T.: Supervised discrete hashing. In: CVPR, pp. 37–45 (2015)

  36. Shen, M., Liu, X., Gong, R., Han, K.: Balanced binary neural networks with gated residual. In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4197–4201. IEEE (2020)

  37. Shen, X., Pan, S., Liu, W., Ong, Y., Sun, Q.: Discrete network embedding. In: IJCAI, pp. 3549–3555 (2018)

  38. Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., Mei, Q.: LINE: Large-scale information network embedding. In: WWW, pp. 1067–1077 (2015)

  39. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention is all you need. In: Neural Information Processing Systems NeurIPS, pp. 5998–6008 (2017)

  40. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903(2017)

  41. Wang, C., Pan, S., Hu, R., Long, G., Jiang, J., Zhang, C.: Attributed graph clustering: a deep attentional embedding approach. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 3670–3676 (2019)

  42. Wang, H., Lian, D., Zhang, Y., Qin, L., Lin, X.: Gognn: Graph of graphs neural network for predicting structured entity interactions. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 1317–1323. ijcai.org (2020)

  43. Wang, J., Zhang, T., Song, J., Sebe, N., Shen, H.T.: A survey on learning to hash. IEEE Trans. Pattern Anal. Mach. Intell. TPAMI. 40(4), 769–790 (2018)

    Article  Google Scholar 

  44. Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8, 229–256 (1992)

    MATH  Google Scholar 

  45. Xu, K., Hu, W., Leskovec, J., Jegelka, S.: How Powerful are Graph Neural Networks?. In: 7Th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 (2019)

  46. Yang, H., Pan, S., Zhang, P., Chen, L., Lian, D., Zhang, C.: Binarized Attributed Network Embedding. In: IEEE ICDM, pp. 1476–1481 (2018)

  47. Zhang, J., Pan, Y., Yao, T., Zhao, H.: Mei, t.: dabnn: a super fast inference framework for binary neural networks on arm devices. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 2272–2275 (2019)

  48. Zhang, M., Chen, Y.: Link prediction based on graph neural networks. In: Neural Information Processing Systems NeurIPS, pp. 5171–5181 (2018)

  49. Zhang, X., Liu, H., Li, Q., Wu, X.: Attributed graph clustering via adaptive graph convolution. In: Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 4327–4333 (2019)

  50. Zhou, S., Ni, Z., Zhou, X., Wen, H., Wu, Y., Zou, Y.: Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR arXiv:abs/1606.06160 (2016)

  51. Zhu, F., Gong, R., Yu, F., Liu, X., Wang, Y., Li, Z., Yang, X., Yan, J.: Towards unified int8 training for convolutional neural network. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition CVPR, pp. 1969–1979 (2020)

Download references

Acknowledgements

Defu Lian is supported by grants from the National Natural Science Foundation of China (No. 61976198 and 62022077). Ying Zhang is supported by ARC FT170100128 and DP180103096. Lu Qin is supported by ARC FT200100787.

Author information

Authors and Affiliations

  1. CAI, University of Technology Sydney, Sydney, Australia

    Hanchen Wang, Ying Zhang & Lu Qin

  2. University of Science and Technology of China, Anhui, China

    Defu Lian

  3. University of Technology Sydney, Sydney, Australia

    Xiangjian He & Yiguang Lin

  4. University of New South Wales, Sydney, Australia

    Xuemin Lin

Authors
  1. Hanchen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Defu Lian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ying Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lu Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiangjian He
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yiguang Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xuemin Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Defu Lian.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, H., Lian, D., Zhang, Y. et al. Binarized graph neural network. World Wide Web 24, 825–848 (2021). https://doi.org/10.1007/s11280-021-00878-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-021-00878-3

Keywords

Search

Navigation