Skip to main content
SpringerLink
Log in

Computing semantic similarity of texts by utilizing dependency graph

  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

The problem of Semantic Textual Similarity (STS) is a significant issue in Natural Language Processing (NLP). STS recognizes and measures semantic relations between two texts. Since the ability to determine the degree of the semantic relationship between sentence pairs is an integral part of machines that understand and infer natural language, we intend to improve the performance of the neural network systems computing the degree of the semantic relation. We propose a graph-U-Net model that operates on a dependency graph and is placed on top of a transformer. Our proposed model indicates the importance of the words in the sentence by assigning the words to several levels while a score as a degree of importance is computed for each level. These scores are used as a weighted average to produce the final result. The importance of the words is new information that our proposed model extracts from the STS and Paraphrase Identification (PI) datasets. We examine the effect of the proposed model on the performance of some transformers in computing semantic relation scores. We use STS2017 and MRPC datasets to evaluate our proposed model. Experimental evaluations show that compared to the transformers, our proposed model obtains a higher value of Pearson and Spearman correlation coefficients and also generates valuable representations for each input so that they improve the Pearson and Spearman values of the systems computing the degree of semantic equivalence between two texts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Data availability

The datasets analysed during the current study are available through:

STS2017: http://ixa2.si.ehu.eus/stswiki/index.php/STSbenchmark

MRPC: https://metatext.io/datasets/microsoft-research-paraphrase-corpus-(mrpc).

References

  • Bastings, J., Titov, I., Aziz, W., et al. (2017). Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 1957–1967). Presented at the EMNLP 2017. Association for Computational Linguistics. https://doi.org/10.18653/v1/D17-1209

  • Bowman, S. R., Vilnis, L., Vinyals, O., et al. (2016). Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL conference on computational natural language learning (pp. 10–21). Presented at the CoNLL 2016. Association for Computational Linguistics. https://doi.org/10.18653/v1/K16-1002

  • Cer, D., Diab, M., Agirre, E., et al. (2017). SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017) (pp. 1–14). Presented at the SemEval 2017. Association for Computational Linguistics. https://doi.org/10.18653/v1/S17-2001

  • Conneau, A., & Lample, G. (2019). Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 32). Curran Associates, Inc. https://proceedings.neurips.cc/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf. Accessed 20 Jan 2022.

  • Conneau, A., Kiela, D., Schwenk, H., et al. (2017). Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 670–680). Association for Computational Linguistics. https://doi.org/10.18653/v1/D17-1070

  • Dolan, B., & Brockett, C. (2005). Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005) (Third International Workshop on Paraphrasing (IWP2005)). Asia Federation of Natural Language Processing. https://www.microsoft.com/en-us/research/publication/automatically-constructing-a-corpus-of-sentential-paraphrases/. Accessed 6 Feb 2022.

  • Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT (pp. 4171–4186). Presented at the 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota. https://aclanthology.org/N19-1423.pdf. Accessed 20 Jan 2022. 

  • Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., et al. (2015). Convolutional networks on graphs for learning molecular fingerprints. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 28). Curran Associates, Inc. https://proceedings.neurips.cc/paper/2015/file/f9be311e65d81a9ad8150a60844bb94c-Paper.pdf. Accessed 30 Jan 2022.

  • Gao, H., & Ji, S. (2021). Graph U-Nets. IEEE transactions on pattern analysis and machine intelligence. Presented at the IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2021.3081010

  • Girshick, R. (2015). Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) (pp. 1440–1448).

  • He, H., & Lin, J. (2016). Pairwise word interaction modeling with deep neural networks for semantic similarity measurement. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies (pp. 937–948). Presented at the NAACL-HLT 2016. Association for Computational Linguistics. https://doi.org/10.18653/v1/N16-1108

  • He, H., Gimpel, K., & Lin, J. (2015). Multi-perspective sentence similarity modeling with convolutional neural networks. In Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 1576–1586). Presented at the EMNLP 2015, Lisbon, Portugal: Association for Computational Linguistics. https://doi.org/10.18653/v1/D15-1181

  • Iyyer, M., Manjunatha, V., Boyd-Graber, J., & Daumé III, H. (2015). Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (Volume 1: Long Papers) (pp. 1681–1691). Presented at the ACL-IJCNLP 2015. Association for Computational Linguistics. https://doi.org/10.3115/v1/P15-1162

  • Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. Presented at the International Conference on Learning Representations (ICLR) 2014, Banff, Canada. https://openreview.net/forum?id=33X9fd2-9FyZd. Accessed 20 Jan 2022

  • Kipf, T. N., & Welling, M. (2017). Semi-supervised classification with graph convolutional networks. In ICLR 2017. Presented at the 5th International Conference on Learning Representations, Palais des Congrès Neptune, Toulon, France. Accessed 21 Jan 2022

  • Lan, Z., Chen, M., Goodman, S., et al. (2020). ALBERT: A Lite BERT for self-supervised learning of language representations. arXiv:1909.11942 [cs]. http://arxiv.org/abs/1909.11942. Accessed 5 Jan 2022

  • Liu, Y., Ott, M., Goyal, N., et al. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692 [cs]. http://arxiv.org/abs/1907.11692. Accessed 17 Feb 2022

  • Manning, C. D., Surdeanu, M., Bauer, J., et al. (2014). The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations (pp. 55–60). http://www.aclweb.org/anthology/P/P14/P14-5010. Accessed 25 Jan 2022.

  • Marcheggiani, D., & Titov, I. (2017). Encoding sentences with graph convolutional networks for semantic role labeling. In Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 1506–1515). Presented at the EMNLP 2017. Association for Computational Linguistics. https://doi.org/10.18653/v1/D17-1159

  • Marelli, M., Bentivogli, L., Baroni, M., et al. (2014). SemEval-2014 task 1: evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014) (pp. 1–8). Presented at the SemEval 2014. Association for Computational Linguistics. https://doi.org/10.3115/v1/S14-2001

  • Morishita, M., Oda, Y., Neubig, G., et al. (2017). An empirical study of mini-batch creation strategies for neural machine translation. In Proceedings of the first workshop on neural machine translation (pp. 61–68). Association for Computational Linguistics. https://doi.org/10.18653/v1/W17-3208

  • Mueller, J., & Thyagarajan, A. (2016). Siamese recurrent architectures for learning sentence similarity. In Proceedings of the thirtieth AAAI conference on artificial intelligence (pp. 2786–2792). AAAI Press. https://doi.org/10.5555/3016100.3016291

  • Pennington, J., Socher, R., & Manning, C. D. (2014). GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP) (pp. 1532–1543). http://www.aclweb.org/anthology/D14-1162. Accessed 22 Jan 2020. 

  • PyTorch Geometric. (n.d.). GitHub. https://github.com/rusty1s/pytorch_geometric. Accessed 20 Jan 2022.

  • Rocktäschel, T., Grefenstette, E., Hermann, K. M., et al. (2015). Reasoning about entailment with neural attention. arXiv.org. https://arxiv.org/abs/1509.06664v4. Accessed 24 July 2021

  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In N. Navab, J. Hornegger, W. M. Wells, & A. F. Frangi (Eds.), Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015 (pp. 234–241). Springer International Publishing.

    Chapter  Google Scholar 

  • Sennrich, R., Haddow, B., & Birch, A. (2016). Neural machine translation of rare words with subword units. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers) (pp. 1715–1725). Association for Computational Linguistics. https://doi.org/10.18653/v1/P16-1162

  • Socher, R., Huang, E. H., Pennington, J., et al. (2011). Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proceedings of the 24th international conference on neural information processing systems (pp. 801–809). Curran Associates Inc. Accessed 23 Oct 2022

  • Stanford CoreNLP. (n.d.). GitHub. https://github.com/stanfordnlp/CoreNLP. Accessed 20 July 2021

  • Tai, K. S., Socher, R., & Manning, C. D. (2015). Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers) (pp. 1556–1566). Presented at the ACL-IJCNLP 2015. Association for Computational Linguistics. https://doi.org/10.3115/v1/P15-1150

  • Tarnowska, K. A., & Ras, Z. W. (2019). Sentiment analysis of customer data. Web Intelligence, 17(4), 343–363. https://doi.org/10.3233/WEB-190423

    Article  Google Scholar 

  • Tarnowska, K. A., & Ras, Z. W. (2021). NLP-Based Customer Loyalty Improvement Recommender System (CLIRS2). Big Data and Cognitive Computing, 5(1), 4. https://doi.org/10.3390/bdcc5010004

    Article  Google Scholar 

  • Tien, N. H., Le, N. M., Tomohiro, Y., & Tatsuya, I. (2019). Sentence modeling via multiple word embeddings and multi-level comparison for semantic textual similarity. Information Processing & Management, 56(6), 102090. https://doi.org/10.1016/j.ipm.2019.102090

    Article  Google Scholar 

  • Transformers. (n.d.). Transformers.. https://huggingface.co/transformers/v2.9.1/. Accessed 20 Aug 2021

  • Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 30). Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf. Accessed 8 Jan 2019

  • Wang, A., Singh, A., Michael, J., et al. (2018). GLUE: a multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP (pp. 353–355). Association for Computational Linguistics. https://doi.org/10.18653/v1/W18-5446

  • Williams, A., Nangia, N., & Bowman, S. (2018). A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: Human language technologies, volume 1 (long papers) (pp. 1112–1122). Association for Computational Linguistics. http://aclweb.org/anthology/N18-1101. Accessed 20 Jan 2022

  • Yang, Y., Yuan, S., Cer, D., et al. (2018). Learning semantic textual similarity from conversations. In Proceedings of the third workshop on representation learning for NLP (pp. 164–174). Association for Computational Linguistics. https://doi.org/10.18653/v1/W18-3022

  • Yang, Z., Dai, Z., Yang, Y., et al. (2019). XLNet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 32). Curran Associates, Inc. https://proceedings.neurips.cc/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-Paper.pdf. Accessed 20 Aug 2021.

  • Žagar, A., & Robnik-Šikonja, M. (2022). Cross-lingual transfer of abstractive summarizer to less-resource language. Journal of Intelligent Information Systems, 58(1), 153–173. https://doi.org/10.1007/s10844-021-00663-8

    Article  Google Scholar 

  • Zhang, X., Yang, Y., Yuan, S., et al. (2019). Syntax-infused variational autoencoder for text generation. In Proceedings of the 57th annual meeting of the association for computational linguistics (pp. 2069–2078). Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1199

  • Zhou, Y., Liu, C., & Pan, Y. (2016). Modelling sentence pairs with tree-structured attentive encoder. In Proceedings of COLING 2016, the 26th international conference on computational linguistics: Technical papers (pp. 2912–2922). https://aclanthology.org/C16-1274. Accessed 20 Jan 2022.

Download references

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Faculty of Electrical and Computer Engineering, University of Tabriz, Tabriz, 51666-16471, Iran

    Majid Mohebbi, Seyed Naser Razavi & Mohammad Ali Balafar

Authors
  1. Majid Mohebbi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Seyed Naser Razavi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammad Ali Balafar
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, all authors; methodology, M.M.; experimental designs, all authors; model development and writing – original draft preparation, M.M.; writing – review and editing, all authors.

Corresponding author

Correspondence to Majid Mohebbi.

Ethics declarations

Ethical approval and consent to participate

Not Applicable.

Consent for publication

Not Applicable.

Human and animal ethics

Not Applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mohebbi, M., Razavi, S.N. & Balafar, M.A. Computing semantic similarity of texts by utilizing dependency graph. J Intell Inf Syst 61, 421–452 (2023). https://doi.org/10.1007/s10844-022-00771-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-022-00771-z

Keywords

Search

Navigation