Skip to main content
SpringerLink
Log in

A contrastive knowledge graph embedding model with hierarchical attention and dynamic completion

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Recently, multi-head Graph Attention Networks (GATs) have achieved satisfactory performance in Knowledge Graph Embedding (KGE) tasks by imposing attention mechanism in local information. However, existing GATs based KGE approaches update entities with few neighbors is difficult to obtain structured semantic information, and these methods only use relations to model the local pairwise importance of entities, which result in missing semantic information of entity embedding. Meanwhile, different entities may have the same position in vector space, which result in poor performance of the model. To this end, we propose a contrastive knowledge graph embedding model named HADC with hierarchical attention network and dynamic completion. HADC dynamically adds the neighbors of entities to complement its local structural information, incorporates both entities’ and relations’ importance in any given entity’s neighborhood, and proposes a contrastive learning-based loss function to distinguish the position of positive and negative samples in vector space. Different experiments on three standard datasets confirm the effectiveness of our innovations, and the performance of our proposed HADC is significantly improved compared to the state-of-the-art methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability statement

All data used during this study are available in the https://github.com/MiracleDesigner/data.

References

  1. He H, Balakrishnan A, Eric M, Liang P (2017) Learning symmetric collaborative dialogue agents with dynamic knowledge graph embeddings. arXiv preprint arXiv:1704.07130

  2. Keizer S, Guhe M, Cuayáhuitl H, Efstathiou I, Engelbrecht K-P, Dobre M, Lascarides A, Lemon O et al (2017) Evaluating persuasion strategies and deep reinforcement learning methods for negotiation dialogue agents. ACL

  3. Berant J, Chou A, Frostig R, Liang P (2013) Semantic parsing on freebase from question-answer pairs. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533–1544

  4. Berant J, Liang P (2014) Semantic parsing via paraphrasing. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1415–1425

  5. Zhang Y, Liu K, He S, Ji G, Liu Z, Wu H, Zhao J (2016) Question answering over knowledge base with neural attention combining global knowledge information. arXiv preprint arXiv:1606.00979

  6. Diefenbach D, Singh K, Maret P (2018) Wdaqua-core1: a question answering service for rdf knowledge bases. In: Companion Proceedings of the The Web Conference 2018, pp. 1087–1091

  7. Lukasova A, Zacek M, Vajgl M (2012) Reasoning in graph-based clausal form logic. Int J Comput Sci Issues (IJCSI) 9(1):37

    Google Scholar 

  8. Lukasová A, Vajgl M, Zacek M (2016) Knowledge represented using rdf semantic network in the concept of semantic web. In: AIP Conference Proceedings, vol. 1738, p. 120012. AIP Publishing LLC

  9. Yang B, Yih SW-t, He X, Gao J, Deng L (2015) Embedding entities and relations for learning and inference in knowledge bases. In: Proceedings of the International Conference on Learning Representations (ICLR) 2015

  10. Dettmers T, Minervini P, Stenetorp P, Riedel S (2018) Convolutional 2d knowledge graph embeddings. In: Thirty-second AAAI Conference on Artificial Intelligence

  11. Nathani D, Chauhan J, Sharma C, Kaul M (2019) Learning attention-based embeddings for relation prediction in knowledge graphs. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4710–4723

  12. Zhang Z, Zhuang F, Zhu H, Shi Z, Xiong H, He Q (2020) Relational graph neural network with hierarchical attention for knowledge graph completion. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9612–9619

  13. Zhao Y, Zhou H, Xie R, Zhuang F, Li Q, Liu J (2021) Incorporating global information in local attention for knowledge representation learning. In: Findings of the Association for Computational Linguistics: ACL-IJCNLP, pp. 1341–1351

  14. Bordes A, Usunier N, Garcia-Durán A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. In: Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 2, pp. 2787–2795

  15. Lin Y, Liu Z, Sun M, Liu Y, Zhu X (2015) Learning entity and relation embeddings for knowledge graph completion. In: Twenty-ninth AAAI Conference on Artificial Intelligence

  16. Wang Z, Zhang J, Feng J, Chen Z (2014) Knowledge graph embedding by translating on hyperplanes. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 28

  17. Ji G, He S, Xu L, Liu K, Zhao J (2015) Knowledge graph embedding via dynamic mapping matrix. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 687–696

  18. Xiao H, Huang M, Zhu X (2016) Transg: A generative model for knowledge graph embedding. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2316–2325

  19. Trouillon T, Welbl J, Riedel S, Gaussier É, Bouchard G (20160 Complex embeddings for simple link prediction. In: International Conference on Machine Learning, pp. 2071–2080 PMLR

  20. Sun Z, Deng Z-H, Nie J-Y, Tang J (2018) Rotate: Knowledge graph embedding by relational rotation in complex space. In: International Conference on Learning Representations

  21. Balkir E, Naslidnyk M, Palfrey D, Mittal A (2019) Using pairwise occurrence information to improve knowledge graph completion on large-scale datasets. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3591–3596

  22. Huang X, Tang J, Tan Z, Zeng W, Wang J, Zhao X (2021) Knowledge graph embedding by relational and entity rotation. Knowl-Based Syst 229:107310

    Article  Google Scholar 

  23. Socher R, Chen D, Manning CD, Ng A (2013) Reasoning with neural tensor networks for knowledge base completion. In: Advances in Neural Information Processing Systems, pp. 926–934

  24. Nickel M, Tresp V, Kriegel H-P (2011) A three-way model for collective learning on multi-relational data. In: Proceedings of the 28th International Conference on International Conference on Machine Learning, pp. 809–816

  25. Nickel M, Rosasco L, Poggio T (2016) Holographic embeddings of knowledge graphs. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30

  26. Balažević I, Allen C, Hospedales T (2019) Tucker: Tensor factorization for knowledge graph completion. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5185–5194

  27. Nguyen TD, Nguyen DQ, Phung D et al (2018) A novel embedding model for knowledge base completion based on convolutional neural network. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 327–333

  28. Vashishth S, Sanyal S, Nitin V, Agrawal N, Talukdar P (2020) Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. In: Proceedings of the AAAI Conference on Artificial Intelligence, 34:3009–3016

  29. Xie Z, Zhou G, Liu J, Huang X (2020) Reinceptione: Relation-aware inception network with joint local-global structural information for knowledge graph embedding. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5929–5939

  30. Che F, Zhang D, Tao J, Niu M, Zhao B (2020) Parame: Regarding neural network parameters as relation embeddings for knowledge graph completion. In: Proceedings of the AAAI Conference on Artificial Intelligence, 34:2774–2781

  31. Vashishth S, Sanyal S, Nitin V, Talukdar P (2019) Composition-based multi-relational graph convolutional networks. In: International Conference on Learning Representations

  32. Bansal T, Juan D-C, Ravi S, McCallum A (2019) A2n: Attending to neighbors for knowledge graph inference. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4387–4392

  33. Schlichtkrull M, Kipf TN, Bloem P, Van Den Berg R, Titov I, Welling M (2018) Modeling relational data with graph convolutional networks. In: European Semantic Web Conference, pp. 593–607. Springer

  34. Chami I, Wolf A, Juan D-C, Sala F, Ravi S, Ré C (2020) Low-dimensional hyperbolic knowledge graph embeddings. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6901–6914

  35. Chen S, Liu X, Gao J, Jiao J, Zhang R, Ji Y (2021) Hitter: Hierarchical transformers for knowledge graph embeddings. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10395–10407

  36. Bai Y, Ying Z, Ren H, Leskovec J (2021) Modeling heterogeneous hierarchies with relation-specific hyperbolic cones. Adv Neural Inf Process Syst 34:12316–12327

    Google Scholar 

  37. Bahdanau D, Cho KH, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations, ICLR 2015

  38. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008

  39. Veličković P, Cucurull G, Casanova A, Romero A, Liò P, Bengio Y (2018) Graph attention networks. In: International Conference on Learning Representations

  40. Qian W, Fu C, Zhu Y, Cai D, He X (2018) Translating embeddings for knowledge graph completion with relation attention mechanism. In: IJCAI, pp. 4286–4292

  41. Lu Y-J, Li C-T (2020) Gcan: Graph-aware co-attention networks for explainable fake news detection on social media. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 505–514

  42. Wu F, Souza A, Zhang T, Fifty C, Yu T, Weinberger K (2019) Simplifying graph convolutional networks. In: International Conference on Machine Learning, pp. 6861–6871. PMLR

  43. Xu K, Hu W, Leskovec J, Jegelka S (2018) How powerful are graph neural networks? In: International Conference on Learning Representations

  44. Thekumparampil KK, Wang C, Oh S, Li L-J (2018) Attention-based graph neural network for semi-supervised learning. arXiv preprint arXiv:1803.03735

  45. Lee JB, Rossi RA, Kim S, Ahmed NK, Koh E (2019) Attention models in graphs: a survey. ACM Transa Knowl Discov from Data (TKDD) 13(6):1–25

    Article  Google Scholar 

  46. Yang Y, Wang X, Song M, Yuan J, Tao D (2019) Spagan: Shortest path graph attention network. In: IJCAI

  47. Han X, Liu Z, Sun M (2018) Neural knowledge acquisition via mutual attention between knowledge graph and text. In: Thirty-second AAAI Conference on Artificial Intelligence

  48. Busbridge D, Sherburn D, Cavallo P, Hammerla NY (2019) Relational graph attention networks. arXiv preprint arXiv:1904.05811

  49. Li Z, Zhao Y, Zhang Y, Zhang Z (2022) Multi-relational graph attention networks for knowledge graph completion. Knowl-Based Syst 251:109262

    Article  Google Scholar 

  50. Chen T, Kornblith S, Norouzi M, Hinton G (2020) A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR

  51. Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

  52. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826

  53. Toutanova K, Chen D, Pantel P, Poon H, Choudhury P, Gamon M (2015) Representing text for joint embedding of text and knowledge bases. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1499–1509

  54. Xiong W, Hoang T, Wang WY (2017) Deeppath: A reinforcement learning method for knowledge graph reasoning. arXiv preprint arXiv:1707.06690

  55. Bollacker K, Evans C, Paritosh P, Sturge T, Taylor J (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, pp. 1247–1250

  56. Miller GA (1995) Wordnet: a lexical database for english. Commun ACM 38(11):39–41

    Article  Google Scholar 

  57. Tang Z, Pei S, Zhang Z, Zhu Y, Zhuang F, Hoehndorf R, Zhang X (2022) Positive-unlabeled learning with adversarial data augmentation for knowledge graph completion. In: International Joint Conferences on Artificial Intelligence

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (No.62192781).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710049, Shaanxi, China

    Bin Shang, Yinliang Zhao, Jun Liu, Yifan Liu & Chenxin Wang

  2. National Engineering Lab for Big Data Analytics, Xi’an, 710049, Shaanxi, China

    Yinliang Zhao & Jun Liu

Authors
  1. Bin Shang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yinliang Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yifan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chenxin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yinliang Zhao.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shang, B., Zhao, Y., Liu, J. et al. A contrastive knowledge graph embedding model with hierarchical attention and dynamic completion. Neural Comput & Applic 35, 15005–15018 (2023). https://doi.org/10.1007/s00521-023-08514-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08514-z

Keywords

Search

Navigation