Skip to main content
SpringerLink
Log in

ReLMKG: reasoning with pre-trained language models and knowledge graphs for complex question answering

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

The goal of complex question answering over knowledge bases (KBQA) is to find an answer entity in a knowledge graph. Recent information retrieval-based methods have focused on the topology of the knowledge graph, ignoring inconsistencies between knowledge graph embeddings and natural language embeddings, and cannot effectively utilize both implicit and explicit knowledge for reasoning. In this paper, we propose a novel model, ReLMKG, to address this challenge. This approach performs joint reasoning on a pre-trained language model and the associated knowledge graph. The complex question and textual paths are encoded by the language model, bridging the gap between the question and the knowledge graph and exploiting implicit knowledge without introducing additional unstructured text. The outputs of different layers in the language model are used as instructions to guide a graph neural network to perform message propagation and aggregation in a step-by-step manner, which utilizes the explicit knowledge contained in the structured knowledge graph. We analyse the reasoning ability of the ReLMKG model for knowledge graphs with different degrees of sparseness and evaluate the generalizability of the model. Experiments conducted on the Complex WebQuestions and WebQuestionsSP datasets demonstrate the effectiveness of our approach on KBQA tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Fig. 4
Algorithm 2
Algorithm 3
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The data generated and analysed during the current study are available from the corresponding author upon reasonable request.

Code Availability

Some or all models, code that support the findings of this study are available from the corresponding author upon reasonable request.

Notes

  1. https://huggingface.co/bert-base-uncased

References

  1. Yang Z, Qi P, Zhang S, Bengio Y, Cohen W, Salakhutdinov R, Manning CD (2018) Hotpotqa: a dataset for diverse, explainable multi-hop question answering. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 2369–2380

  2. Etezadi R, Shamsfard M (2022) The state of the art in open domain complex question answering: a survey. Appl Intell, 1–21

  3. Fang Y, Sun S, Gan Z, Pillai R, Wang S, Liu J (2020) Hierarchical graph network for multi-hop question answering. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 8823–8838

  4. Jiang Y, Bansal M (2019) Avoiding reasoning shortcuts: adversarial evaluation, training, and model development for multi-hop qa. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 2726–2736

  5. Cao X, Liu Y (2021) Coarse-grained decomposition and fine-grained interaction for multi-hop question answering. J Intell Inf Syst, 1–21

  6. Zhou M, Huang M, Zhu X (2018) An interpretable reasoning network for multi-relation question answering. In: Proceedings of the 27th international conference on computational linguistics, pp 2010–2022

  7. Cao X, Liu Y, Hu B, Zhang Y (2021) Dual-channel reasoning model for complex question answering. Complexity, 2021

  8. Lan Y, He G, Jiang J, Jiang J, Zhao WX, Wen J-R (2021) A survey on complex knowledge base question answering: Methods, challenges and solutions. In: Proceedings of the thirtieth international joint conference on artificial intelligence, IJCAI-21

  9. Yen A-Z, Huang H-H, Chen H-H (2021) Unanswerable question correction in question answering over personal knowledge base. In: Thirty-Fifth AAAI conference on artificial intelligence (AAAI-21)

  10. Kapanipathi P, Abdelaziz I, Ravishankar S, Roukos S, Gray A, Astudillo RF, Chang M, Cornelio C, Dana S, Fokoue-Nkoutche A et al (2021) Leveraging abstract meaning representation for knowledge base question answering. In: Findings of the association for computational linguistics: ACL-IJCNLP 2021, pp 3884–3894

  11. Saxena A, Chakrabarti S, Talukdar P (2021) Question answering over temporal knowledge graphs. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pp 6663–6676

  12. Auer S, Bizer C, Kobilarov G, Lehmann J, Cyganiak R, Ives Z (2007) Dbpedia: a nucleus for a web of open data. In: The semantic web, pp 722–735

  13. Suchanek FM, Kasneci G, Weikum G (2007) Yago: a core of semantic knowledge. In: Proceedings of the 16th international conference on world wide web, pp 697–706

  14. Bollacker K, Evans C, Paritosh P, Sturge T, Taylor J (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD international conference on management of data, pp 1247–1250

  15. Qiu Y, Wang Y, Jin X, Zhang K (2020) Stepwise reasoning for multi-relation question answering over knowledge graph with weak supervision. In: Proceedings of the 13th international conference on web search and data mining, pp 474–482

  16. He G, Lan Y, Jiang J, Zhao WX, Wen J-R (2021) Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In: Proceedings of the 14th ACM international conference on web search and data mining, pp 553–561

  17. Han J, Cheng B, Wang X (2020) Open domain question answering based on text enhanced knowledge graph with hyperedge infusion Findings of the association for computational linguistics: EMNLP 2020, pp 1475–1481

  18. Sun H, Dhingra B, Zaheer M, Mazaitis K, Salakhutdinov R, Cohen WW Open domain question answering using early fusion of knowledge bases and text

  19. Jiang D, Li G, Sun Y, Kong J, Tao B, Chen D (2019) Grip strength forecast and rehabilitative guidance based on adaptive neural fuzzy inference system using semg. Pers Ubiquit Comput, 1–10

  20. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neur Comput 9(8):1735–1780

    Article  Google Scholar 

  21. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 Workshop on deep learning, December 2014

  22. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. Adv Neural Inform Process Syst, 30

  23. Han J, Cheng B, Wang X (2021) Two-phase hypergraph based reasoning with dynamic relations for multi-hop kbqa. In: Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pp 3615–3621

  24. Shen Y, Yang M, Li Y, Wang D, Zheng H, Chen D (2021) Knowledge-based reasoning network for relation detection. IEEE Transactions on Neural Networks and Learning Systems

  25. Bordes A, Usunier N, Garcia-Duran A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. Advances in Neural Information Processing Systems, 26

  26. Wang Z, Zhang J, Feng J, Chen Z (2014) Knowledge graph embedding by translating on hyperplanes. In: Proceedings of the AAAI conference on artificial intelligence, vol 28

  27. Lin Y, Liu Z, Sun M, Liu Y, Zhu X (2015) Learning entity and relation embeddings for knowledge graph completion. In: Twenty-ninth AAAI conference on artificial intelligence

  28. Peng Y, Zhang J (2020) Lineare: simple but powerful knowledge graph embedding for link prediction. In: 2020 IEEE International conference on data mining (ICDM). IEEE, pp 422–431

  29. Pennington J, Socher R, Manning CD (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp 1532–1543

  30. Lin BY, Chen X, Chen J, Ren X (2019) Kagnet: knowledge-aware graph networks for commonsense reasoning. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 2829–2839

  31. Yasunaga M, Ren H, Bosselut A, Liang P, Leskovec J (2021) Qa-gnn: reasoning with language models and knowledge graphs for question answering. In: Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies, pp 535–546

  32. Molchanov D, Ashukha A, Vetrov D (2017) Variational dropout sparsifies deep neural networks. In: International Conference on Machine Learning. PMLR, pp 2498-2507

  33. Xiong W, Yu M, Chang S, Guo X, Wang WY (2019) Improving question answering over incomplete kbs with knowledge-aware reader. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 4258–4264

  34. Qiu Y, Zhang K, Wang Y, Jin X, Bai L, Guan S, Cheng X (2020) Hierarchical query graph generation for complex question answering over knowledge graph. In: Proceedings of the 29th ACM international conference on information & knowledge management, pp 1285–1294

  35. Sun H, Bedrax-Weiss T, Cohen W (2019) Pullnet: open domain question answering with iterative retrieval on knowledge bases and text. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 2380–2390

  36. Yan Y, Li R, Wang S, Zhang H, Daoguang Z, Zhang F, Wu W, Xu W (2021) Large-scale relation learning for question answering over knowledge bases with pre-trained language models. In: Proceedings of the 2021 conference on empirical methods in natural language processing, pp 3653–3660

  37. Devlin J, Chang M -W, Lee K, Toutanova K (2019) Bert: ptraining of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp 4171–4186

  38. Yu M, Li G, Jiang D, Jiang G, Tao B, Chen D (2019) Hand medical monitoring system based on machine learning and optimal emg feature set. Pers Ubiquit Comput, 1–17

  39. Liu X, Jiang D, Tao B, Jiang G, Sun Y, Kong J, Tong X, Zhao G, Chen B (2021) Genetic algorithm-based trajectory optimization for digital twin robots. Frontiers in Bioengineering and Biotechnology, 9

  40. Haveliwala TH (2003) Topic-sensitive pagerank: a context-sensitive ranking algorithm for web search. IEEE Transa Knowl Data Eng 15(4):784–796

    Article  Google Scholar 

  41. Zhang Y, Dai H, Kozareva Z, Smola AJ, Song L (2018) Variational reasoning for question answering with knowledge graph. In: Thirty-Second AAAI conference on artificial intelligence

  42. Xu K, Lai Y, Feng Y, Wang Z (2019) Enhancing key-value memory neural networks for knowledge based question answering. In: Proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp 2937–2947

  43. Luo K, Lin F, Luo X, Zhu K (2018) Knowledge base question answering via encoding of complex query graphs. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 2185–2194

  44. Chen Y, Li H, Hua Y, Qi G (2021) Formal query building with query structure prediction for complex question answering over knowledge base. In: Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pp 3751–3758

  45. Zhu S, Cheng X, Su S (2020) Knowledge-based question answering by tree-to-sequence learning. Neurocomputing 372:64–72

    Article  Google Scholar 

  46. Sun Y, Zhang L, Cheng G, Qu Y (2020) Sparqa: skeleton-based semantic parsing for complex questions over knowledge bases. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 8952–8959

  47. Bhutani N, Zheng X, Jagadish H (2019) Learning to answer complex questions over knowledge bases with query composition. In: Proceedings of the 28th ACM international conference on information and knowledge management, pp 739–748

  48. Saxena A, Tripathi A, Talukdar P (2020) Improving multi-hop question answering over knowledge graphs using knowledge base embeddings. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 4498–4507

  49. Trouillon T, Welbl J, Riedel S, Gaussier É, Bouchard G (2016) Complex embeddings for simple link prediction. In: International Conference on Machine Learning. PMLR, pp 2071-2080

  50. Ren H, Hu W, Leskovec J (2019) Query2box: reasoning over knowledge graphs in vector space using box embeddings. In: International conference on learning representations

  51. Ren H, Dai H, Dai B, Chen X, Yasunaga M, Sun H, Schuurmans D, Leskovec J, Zhou D (2021) Lego: latent execution-guided reasoning for multi-hop question answering on knowledge graphs. In: International conference on machine learning. PMLR, pp 89598970

  52. Lan Y, Jiang J (2020) Query graph generation for answering multi-hop complex questions from knowledge bases. Assoc Comput Linguis

  53. Zhang Y, Jin L, Zhang Z, Li X, Liu Q, Wang H (2022) Sf-ann: leveraging structural features with an attention neural network for candidate fact ranking. Appl Intell 52(5):5841–5856

    Article  Google Scholar 

  54. Zhang Q, Weng X, Zhou G, Zhang Y, Huang JX (2022) Arl: an adaptive reinforcement learning framework for complex question answering over knowledge base. Inform Process Manag 59(3):102933

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by the Fundamental Research Funds for the Central Universities (Grant number 2020YJS012).

Funding

This research was funded by the Fundamental Research Funds for the Central Universities (Grant number 2020YJS012).

Author information

Authors and Affiliations

  1. School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing, 100044, China

    Xing Cao & Yun Liu

  2. Key Laboratory of Communication and Information Systems, Beijing Municipal Commission of Education, Beijing, 100044, China

    Xing Cao & Yun Liu

Authors
  1. Xing Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yun Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Xing Cao and Yun Liu designed the study and performed the experiments; Xing Cao performed the experiments, analyzed the data, and wrote the manuscript.

Corresponding author

Correspondence to Yun Liu.

Ethics declarations

Competing interests

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cao, X., Liu, Y. ReLMKG: reasoning with pre-trained language models and knowledge graphs for complex question answering. Appl Intell 53, 12032–12046 (2023). https://doi.org/10.1007/s10489-022-04123-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04123-w

Keywords

Search

Navigation