Skip to main content
SpringerLink
Log in

SGPT: Semantic graphs based pre-training for aspect-based sentiment analysis

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Previous studies show effective of pre-trained language models for sentiment analysis. However, most of these studies ignore the importance of sentimental information for pre-trained models. Therefore, we fully investigate the sentimental information for pre-trained models and enhance pre-trained language models with semantic graphs for sentiment analysis. In particular, we introduce Semantic Graphs based Pre-training(SGPT) using semantic graphs to obtain synonym knowledge for aspect-sentiment pairs and similar aspect/sentiment terms. We then optimize the pre-trained language model with the semantic graphs. Empirical studies on several downstream tasks show that proposed model outperforms strong pre-trained baselines. The results also show the effectiveness of proposed semantic graphs for pre-trained model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Figure 3

Similar content being viewed by others

References

  1. Lai, G., Xie, Q., Liu, H., Yang, Y., Hovy, E.: RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017)

  2. Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100,000+ Questions for Machine Comprehension of Text (2016)

  3. Zhang, L., Wang, S., Liu, B.: Deep learning for sentiment analysis : a survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1253 (2018)

  4. Zeng, Z., Zhou, W., Liu, X., Song, Y.: A variational approach to weakly supervised document-level multi-aspect sentiment classification. In: Conference of the North (2019)

  5. Li, J., Chiu, B., Shang, S., Shao, L.: Neural text segmentation and its application to sentiment analysis. IEEE Trans. Knowl. Data Eng. 34(2), 828–842 (2022). https://doi.org/10.1109/TKDE.2020.2983360

    Article  Google Scholar 

  6. Wang, Z., Du, B., Guo, Y.: Domain adaptation with neural embedding matching. IEEE Trans. Neural Networks Learn. Syst. 31(7), 2387–2397 (2020). https://doi.org/10.1109/TNNLS.2019.2935608

    Article  MathSciNet  Google Scholar 

  7. Wang, Z., Du, B., Tu, W., Zhang, L., Tao, D.: Incorporating distribution matching into uncertainty for multiple kernel active learning. IEEE Trans. Knowl. Data Eng. 33(1), 128–142 (2021). https://doi.org/10.1109/TKDE.2019.2923211

    Article  Google Scholar 

  8. Liu, B.: Sentiment analysis and opinion mining. Synth. Lect. Hum. Lang. Technol 5(1), 1–167 (2012)

    Article  MathSciNet  Google Scholar 

  9. Tian, H., Gao, C., Xiao, X., Liu, H., He, B., Wu, H., Wang, H., Wu, F.: Skep: Sentiment knowledge enhanced pre-training for sentiment analysis. arXiv:2005.05635 (2020)

  10. Ke, P., Ji, H., Liu, S., Zhu, X., Huang, M.: Sentilare: Linguistic knowledge enhanced language representation for sentiment analysis. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6975–6988 (2020)

  11. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692 (2019)

  12. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805 (2018)

  13. Zhang, X., Zhao, J.J., LeCun, Y.: Character-level convolutional networks for text classification. CoRR arXiv:abs/1509.01626 (2015)

  14. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642 (2013)

  15. Maria, P., Dimitris, G., John, P., Harris, P., Ion, A., Suresh, M.: Semeval-2014 task 4: As- pect based sentiment analysis. In: Proceedings of the 8th International Workshop on Semantic Evaluation, pp. 27–35 (2014)

  16. Wiebe, J., Wilson, T., Cardie, C.: Annotating expressions of opinions and emotions in language. Lang. Resour. Eval. 39(2), 165–210 (2005)

    Article  Google Scholar 

  17. Wilson, T.A.: Fine-Grained Subjectivity and Sentiment Analysis: Recognizing the Intensity, Polarity, and Attitudes of Private States. PhD thesis, University of Pittsburgh (2008)

  18. Marasovic, A., Frank, A.: SRL4ORL: Improving opinion role labelling using multi-task learning with semantic role labeling. CoRR arXiv:abs/1711.00768 (2017)

  19. Taboada, M., Brooke, J., Tofiloski, M., Voll, K.D., Stede, M.: Lexicon-based methods for sentiment analysis. Comput. Linguist. 37(2), 267–307 (2011)

    Article  Google Scholar 

  20. Shin, B., Lee, T., Choi, J.D.: Lexicon integrated cnn models with attention for sentiment analysis. Alexandra Balahur Dobrescu, 149–158 (2017)

  21. Vo, D.-T., Zhang, Y.: Target-Dependent Twitter Sentiment Classification with Rich Automatic Features. In: Twenty-Fourth International Joint Conference on Artificial Intelligence (2015)

  22. Li, X., Lam, W.: Deep multi-task learning for aspect term extraction with memory interaction. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2886–2892 (2017)

  23. Gui, L., Hu, J., He, Y., Xu, R., Lu, Q., Du, J.: A question answering approach to emotion cause extraction. arXiv:1708.05482 (2017)

  24. Fan, C., Yan, H., Du, J., Gui, L., Bing, L., Yang, M., Xu, R., Mao, R.: A knowledge regularized hierarchical approach for emotion cause analysis. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5618–5628 (2019)

  25. Turney, P.D.: Thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews. arXiv:cs/0212032 (2002)

  26. Taboada, M., Brooke, J., Tofiloski, M., Voll, K., Stede, M.: Lexicon-based methods for sentiment analysis. Comput. Linguist. 37(2), 267–307 (2011). https://doi.org/10.1162/COLI_a_00049

    Article  Google Scholar 

  27. Bakshi, R.K., Kaur, N., Kaur, R., Kaur, G.: Opinion mining and sentiment analysis. In: 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom), pp 452–455. IEEE (2016)

  28. Agarwal, A., Xie, B., Vovsha, I., Rambow, O., Passonneau, R.J.: Sentiment analysis of twitter data. In: Proceedings of the Workshop on Language in Social Media (LSM 2011), pp. 30–38 (2011)

  29. Shin, B., Lee, T., Choi, J.D.: Lexicon integrated cnn models with attention for sentiment analysis. arXiv:1610.06272 (2016)

  30. Qian, Q., Huang, M., Lei, J., Zhu, X.: Linguistically regularized lstms for sentiment classification. arXiv:1611.03949 (2016)

  31. Zeng, Z., Zhou, W., Liu, X., Song, Y.: A variational approach to weakly supervised document-level multi-aspect sentiment classification. arXiv:1904.05055 (2019)

  32. Yang, Z., Salakhutdinov, R., Cohen, W.W.: Transfer learning for sequence tagging with hierarchical recurrent networks. arXiv:1703.06345 (2017)

  33. Ding, Y., Yu, J., Jiang, J.: Recurrent neural networks with auxiliary labels for cross-domain opinion target extraction. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

  34. Wang, X., Gao, T., Zhu, Z., Liu, Z., Li, J., Tang, J.: Kepler: A unified model for knowledge embedding and pre-trained language representation. arXiv:1911.06136 (2019)

  35. Peters, M.E., Neumann, M., Logan IV, R.L., Schwartz, R., Joshi, V., Singh, S., Smith, N.A.: Knowledge enhanced contextual word representations. arXiv:1909.04164 (2019)

  36. Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., Liu, Q.: Ernie: Enhanced language representation with informative entities. arXiv:1905.07129(2019)

  37. Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., Wang, P.: K-bert: Enabling language representation with knowledge graph. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 2901–2908 (2020)

  38. Wang, R., Tang, D., Duan, N., Wei, Z., Huang, X., Cao, C., Jiang, D., Zhou, M., et al.: K-adapter: Infusing knowledge into pre-trained models with adapters. arXiv:2002.01808 (2020)

Download references

Funding

not applicable

Author information

Authors and Affiliations

  1. Hangzhou, Zhejiang, China

    Qian Yong, Rong Xiao & Haihong Tang

  2. Natural Language Processing Lab, Soochow University, Suzhou, 215006, Jiangsu, China

    Chen Chen & Zhongqing Wang

Authors
  1. Qian Yong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chen Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhongqing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rong Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haihong Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Yong qian, chen chen and Zhongqing Wang wrote the main manuscript text and Yong Qian did all experiments. Chen chen prepared figures 1-3. All authors reviewed the manuscript.

Corresponding author

Correspondence to Qian Yong.

Ethics declarations

Ethics approval

not applicable

Competing interests

not applicable

Additional information

Availability of data and materials

All the data can be found online. SST-2:http://nlp.stanford.edu/sentiment MPQA2:http://mpqa.cs.pitt.edu/corpora/mpqa_corpus/mpqa_corpus_2_0/ Amazon-2:https://snap.stanford.edu/data/web-Amazon.html SRL4ORL:http://alt.qcri.org/semeval2014/task4/data/uploads/ Taobao dataset cannot be provided at present because Alibaba has not yet approved, once approved we will release dataset.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article belongs to the Topical Collection: Special Issue on Spatiotemporal Data Management and Analytics for Recommend Guest Editors: Shuo Shang, Xiangliang Zhang and Panos Kalnis

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yong, Q., Chen, C., Wang, Z. et al. SGPT: Semantic graphs based pre-training for aspect-based sentiment analysis. World Wide Web 26, 2201–2214 (2023). https://doi.org/10.1007/s11280-022-01123-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-022-01123-1

Keywords

Search

Navigation