Skip to main content
SpringerLink
Log in

How Does Bayesian Noisy Self-Supervision Defend Graph Convolutional Networks?

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

In recent years, it has been shown that, compared to other contemporary machine learning models, graph convolutional networks (GCNs) achieve superior performance on the node classification task. However, two potential issues threaten the robustness of GCNs, label scarcity and adversarial attacks. .Intensive studies aim to strengthen the robustness of GCNs from three perspectives, the self-supervision-based method, the adversarial-based method, and the detection-based method. Yet, all of the above-mentioned methods can barely handle both issues simultaneously. In this paper, we hypothesize noisy supervision as a kind of self-supervised learning method and then propose a novel Bayesian graph noisy self-supervision model, namely GraphNS, to address both issues. Extensive experiments demonstrate that GraphNS can significantly enhance node classification against both label scarcity and adversarial attacks. This enhancement proves to be generalized over four classic GCNs and is superior to the competing methods across six public graph datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. https://www.kdd.org/kdd-cup/view/kdd-cup-2016.

References

  1. Bishop CM (2006) Pattern recognition and machine learning. Springer, Berlin

    MATH  Google Scholar 

  2. Bruna J, Zaremba W, Szlam A, LeCun Y (2013) Spectral networks and locally connected networks on graphs. arXiv preprint arXiv:1312.6203

  3. Dai H, Li H, Tian T, Huang X, Wang L, Zhu J, Song L (2018) Adversarial attack on graph structured data. arXiv preprint arXiv:1806.02371

  4. Defferrard M, Bresson X, Vandergheynst P (2016) Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in neural information processing systems, pp 3844–3852

  5. Deng Z, Dong Y, Zhu J (2019) Batch virtual adversarial training for graph convolutional networks. arXiv preprint arXiv:1902.09192

  6. Du B, Xinyao T, Wang Z, Zhang L, Tao D (2018) Robust graph-based semisupervised learning for noisy labeled data via maximum correntropy criterion. IEEE Trans Cybern

  7. Du J, Zhang S, Wu G, Moura JM, Kar S (2017) Topology adaptive graph convolutional networks. arXiv preprint arXiv:1710.10370

  8. Entezari N, Al-Sayouri SA, Darvishzadeh A, Papalexakis EE (2020) All you need is low (rank) defending against adversarial attacks on graphs. In: Proceedings of the 13th International Conference on Web Search and Data Mining

  9. Feng F, He X, Tang J, Chua TS (2019) Graph adversarial training: Dynamically regularizing based on graph structure. IEEE Trans Knowl Data Eng

  10. Feng W, Zhang J, Dong Y, Han Y, Luan H, Xu Q, Yang Q, Kharlamov E, Tang J (2020) Graph random neural network for semi-supervised learning on graphs. In: NeurIPS’20

  11. Galke L, Vagliano I, Scherp A (2019) Can graph neural networks go “online"? an analysis of pretraining and inference. arXiv preprint arXiv:1905.06018

  12. Geisler S, Zügner D, Günnemann S (2020) Reliable graph neural networks via robust aggregation. Adv Neural Inf Process Syst, 33

  13. Goldberger J, Ben-Reuven E (2017) Training deep neural-networks using a noise adaptation layer. International Conference on Learning Representations

  14. Hamilton W, Ying Z, Leskovec J (2017) Inductive representation learning on large graphs. In: Advances in neural information processing systems, pp 1024–1034

  15. Hassani K, Khasahmadi AH (2020) Contrastive multi-view representation learning on graphs. In: International conference on machine learning, PMLR, pp 4116–4126

  16. Hu W, Liu B, Gomes J, Zitnik M, Liang P, Pande V, Leskovec J (2019a) Strategies for pre-training graph neural networks. In: International conference on learning representations

  17. Hu W, Fey M, Zitnik M, Dong Y, Ren H, Liu B, Catasta M, Leskovec J (2020a) Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687

  18. Hu Z, Fan C, Chen T, Chang KW, Sun Y (2019b) Pre-training graph neural networks for generic structural feature extraction. arXiv preprint arXiv:1905.13728

  19. Hu Z, Dong Y, Wang K, Chang KW, Sun Y (2020b) Gpt-gnn: Generative pre-training of graph neural networks. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp 1857–1867

  20. Hwang D, Park J, Kwon S, Kim KM, Ha JW, Kim HJ (2020) Self-supervised auxiliary learning with meta-paths for heterogeneous graphs. arXiv preprint arXiv:2007.08294

  21. Jin H, Zhang X (2019) Latent adversarial training of graph convolution networks. In: ICML workshop on learning and reasoning with graph-structured representations

  22. Jin W, Ma Y, Liu X, Tang X, Wang S, Tang J (2020) Graph structure learning for robust graph neural networks. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp 66–74

  23. Kefato ZT, Girdzijauskas S (2021) Self-supervised graph neural networks without explicit negative sampling. arXiv preprint arXiv:2103.14958

  24. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907

  25. Miller BA, Çamurcu M, Gomez AJ, Chan K, Eliassi-Rad T (2019) Improving robustness to attacks against vertex classification. In: MLG Workshop

  26. Misra I, Lawrence Zitnick C, Mitchell M, Girshick R (2016) Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2930–2939

  27. NT H, Jin CJ, Murata T (2019) Learning graph neural networks with noisy labels. arXiv preprint arXiv:1905.01591

  28. Patrini G, Rozza A, Krishna Menon A, Nock R, Qu L (2017) Making deep neural networks robust to label noise: A loss correction approach. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  29. Qiu J, Chen Q, Dong Y, Zhang J, Yang H, Ding M, Wang K, Tang J (2020) Gcc: Graph contrastive coding for graph neural network pre-training. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp 1150–1160

  30. Qu M, Bengio Y, Tang J (2019) Gmnn: Graph markov neural networks. In: International conference on machine learning, PMLR, pp 5241–5250

  31. Reed S, Lee H, Anguelov D, Szegedy C, Erhan D, Rabinovich A (2014) Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596

  32. Rong Y, Bian Y, Xu T, Xie W, Wei Y, Huang W, Huang J (2020) Self-supervised graph transformer on large-scale molecular data. Adv Neural Inf Process Syst, 33

  33. Sen P, Namata G, Bilgic M, Getoor L, Galligher B, Eliassi-Rad T (2008) Collective classification in network data. AI magazine

  34. Shang J, Ma T, Xiao C, Sun J (2019) Pre-training of graph augmented transformers for medication recommendation. arXiv preprint arXiv:1906.00346

  35. Shchur O, Mumme M, Bojchevski A, Günnemann S (2018) Pitfalls of graph neural network evaluation. Relational Representation Learning Workshop, NeurIPS

    Google Scholar 

  36. Sukhbaatar S, Bruna J, Paluri M, Bourdev L, Fergus R (2014) Training convolutional networks with noisy labels. arXiv preprint arXiv:1406.2080

  37. Sun K, Zhu Z, Lin Z (2019) Multi-stage self-supervised learning for graph convolutional networks. arXiv preprint arXiv:1902.11038

  38. Sun L, Dou Y, Yang C, Wang J, Yu PS, Li B (2018) Adversarial attack and defense on graph data: A survey. arXiv preprint arXiv:1812.10528

  39. Tang X, Li Y, Sun Y, Yao H, Mitra P, Wang S (2020) Transferring robustness for graph neural network against poisoning attacks. In: Proceedings of the 13th international conference on web search and data mining

  40. Tsitsulin A, Mottin D, Karras P, Bronstein A, Müller E (2018) Sgr: Self-supervised spectral graph representation learning. arXiv preprint arXiv:1811.06237

  41. Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2018) Graph attention networks. In: International conference on learning representations

  42. Wang S, Chen Z, Ni J, Yu X, Li Z, Chen H, Yu PS (2019a) Adversarial defense framework for graph neural network. arXiv preprint arXiv:1905.03679

  43. Wang X, Liu X, Hsieh CJ (2019b) Graphdefense: Towards robust graph convolutional networks. arXiv preprint arXiv:1911.04429

  44. Wu F, Zhang T, Souza Jr AHd, Fifty C, Yu T, Weinberger KQ (2019a) Simplifying graph convolutional networks. arXiv preprint arXiv:1902.07153

  45. Wu H, Wang C, Tyshetskiy Y, Docherty A, Lu K, Zhu L (2019b) Adversarial examples on graph data: Deep insights into attack and defense. arXiv preprint arXiv:1903.01610

  46. Xie Y, Xu Z, Zhang J, Wang Z, Ji S (2021) Self-supervised learning of graph neural networks: A unified review. arXiv preprint arXiv:2102.10757

  47. Xu B, Shen H, Cao Q, Qiu Y, Cheng X (2019a) Graph wavelet neural network. arXiv preprint arXiv:1904.07785

  48. Xu K, Hu W, Leskovec J, Jegelka S (2018) How powerful are graph neural networks? arXiv preprint arXiv:1810.00826

  49. Xu K, Chen H, Liu S, Chen PY, Weng TW, Hong M, Lin X (2019b) Topology attack and defense for graph neural networks: an optimization perspective. arXiv preprint arXiv:1906.04214

  50. Yao J, Wu H, Zhang Y, Tsang IW, Sun J (2019) Safeguarded dynamic label regression for noisy supervision. Proc AAAI Conf Artif Intell 33:9103–9110

    Google Scholar 

  51. Yasunaga M, Liang P (2020) Graph-based, self-supervised program repair from diagnostic feedback. In: International conference on machine learning, PMLR, pp 10799–10808

  52. You Y, Chen T, Sui Y, Chen T, Wang Z, Shen Y (2020a) Graph contrastive learning with augmentations. Adv Neural Inf Process Syst, 33

  53. You Y, Chen T, Wang Z, Shen Y (2020b) When does self-supervision help graph convolutional networks? In: International conference on machine learning, PMLR, pp 10871–10880

  54. You Y, Chen T, Shen Y, Wang Z (2021) Graph contrastive learning automated. arXiv preprint arXiv:2106.07594

  55. Zhang A, Ma J (2020) Defensevgae: Defending against adversarial attacks on graph data via a variational graph autoencoder. arXiv preprint arXiv:2006.08900

  56. Zhang Y, Khan S, Coates M (2019a) Comparing and detecting adversarial attacks for graph deep learning. In: Proc. Representation Learning on Graphs and Manifolds Workshop, Int. Conf. Learning Representations, New Orleans, LA, USA

  57. Zhang Y, Pal S, Coates M, Ustebay D (2019) Bayesian graph convolutional neural networks for semi-supervised classification. Proc AAAI Conf Artif Intell 33:5829–5836

    Google Scholar 

  58. Zheng C, Zong B, Cheng W, Song D, Ni J, Yu W, Chen H, Wang W (2020) Robust graph representation learning via neural sparsification. In: International conference on machine learning, PMLR, pp 11458–11468

  59. Zhong JX, Li N, Kong W, Liu S, Li TH, Li G (2019) Graph convolutional label noise cleaner: Train a plug-and-play action classifier for anomaly detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition

  60. Zügner D, Günnemann S (2019) Adversarial attacks on graph neural networks via meta learning. arXiv preprint arXiv:1902.08412

  61. Zügner D, Akbarnejad A, Günnemann S (2018) Adversarial attacks on neural networks for graph data. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 2847–2856

Download references

Author information

Authors and Affiliations

  1. Indiana University-Purdue University Indianapolis, Indianapolis, IN, USA

    Jun Zhuang & Mohammad Al Hasan

Authors
  1. Jun Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Al Hasan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Zhuang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Implementation

Implementation

1.1 Hardware and Software

All above-mentioned experiments are conducted on the server with the following configurations:

  • Operating System: Ubuntu 18.04.5 LTS

  • CPU: Intel(R) Xeon(R) Gold 6258R CPU @ 2.70 GHz

  • GPU: NVIDIA Tesla V100 PCIe 16GB

  • Software: Python 3.8, PyTorch 1.7.

1.2 Model Architecture and Hyper-parameters

The model architecture of GCNs and hyper-parameters are described in Table 6 and 7, respectively. We assume the Dirichlet distribution in this paper is symmetric and thus fix \(\alpha \) as 1.0 (a.k.a. flat Dirichlet distribution).

Table 6 Model architecture of GCNs
Full size table
Table 7 Model hyper-parameters (#Hidden denotes the number of neurons in each hidden layer of GCNs.)
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhuang, J., Hasan, M.A. How Does Bayesian Noisy Self-Supervision Defend Graph Convolutional Networks?. Neural Process Lett 54, 2997–3018 (2022). https://doi.org/10.1007/s11063-022-10750-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-10750-8

Keywords

Search

Navigation