Skip to main content
SpringerLink
Log in

BERT-PG: a two-branch associative feature gated filtering network for aspect sentiment classification

  • Research
  • Published:
Journal of Intelligent Information Systems Aims and scope Submit manuscript

Abstract

Aspect sentiment classification is an important branch of sentiment classification that has gained increasing attention recently. Existing aspect sentiment classification methods typically use different network branches to encode context and aspect words separately, and then use an attention mechanism to capture their associations. This attention-based approach cannot completely ignore the contexts unrelated to the current aspect words, which brings noise interference. In this paper, a gated filtering network based on BERT is suggested as a solution to this issue. We employ BERT to encode the text semantics of contexts and sentence pairs consisting of context and aspect words respectively, and to extract lexical features as well as associative features of context and aspect words. Based on this, we designed a gating module that, unlike the attention mechanism, uses association features to precisely filter irrelevant contexts. Additionally, because the BERT network parameters are so big, there is a tendency to over-fitting during training. To effectively combat this problem, we developed a loss function with a threshold. We carried out extensive experiments using three benchmark datasets to verify the performance of our proposed model. The experimental results show that the method improves the accuracy by 0.5%, 1.39% and 2.57% on the Laptop, Restaurant and Twitter datasets respectively, and 1.564%, 2.36% and 4.144% on Macro-F1 respectively, compared to the recent RA-CNN (BERT), proving that our method is effective in improving the presentation of aspect sentiment classification in comparison to other cutting-edge sentiment classification methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data availability

The datasets generated during and/or analysed during the current study are available from the corresponding author on reasonable request.

References

  • Abel, J., & Lantow, B. (2019). A methodological framework for dictionary and rule-based text classification. In: 11th International Conference on Knowledge Discovery and Information Retrieval, pp. 330–337. SciTePress. https://doi.org/10.5220/0008121503300337

  • Chen, P., Sun, Z., Bing, L., et al. (2017). Recurrent attention network on memory for aspect sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 452–461. Association for Computational Linguistics. https://doi.org/10.18653/v1/D17-1047

  • Chung, J., Gulcehre, C., Cho, K., et al. (2014). Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NIPS 2014 Workshop on Deep Learning, pp. 1–9. MIT Press. https://doi.org/10.48550/arXiv.1412.3555

  • Colbrook, M. J., Antun, V., & Hansen, A. C. (2022). The difficulty of computing stable and accurate neural networks: On the barriers of deep learning and smale’s 18th problem. Proceedings of the National Academy of Sciences of the United States of America, 119, 2107151119. https://doi.org/10.1073/pnas.2107151119

    Article  MathSciNet  Google Scholar 

  • Dauphin, Y. N., Fan, A., Auli, M., et al. (2017). Language modeling with gated convolutional networks. In: Proceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 933–941. JMLR.org. https://doi.org/10.48550/arXiv.1612.08083

  • Dong, L., Wei, F., Tan, C., et al. (2014). Adaptive recursive neural network for target-dependent twitter sentiment classification. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, vol. 2, pp. 49–54. Association for Computational Linguistics. https://doi.org/10.3115/v1/P14-2009

  • Gan, S., Shao, S., Chen, L., et al. (2021). Adapting hidden naive bayes for text classification. Mathematics, 9, 2378. https://doi.org/10.3390/math9192378

    Article  Google Scholar 

  • Gong, C., Du, Y., Li, X., et al. (2020). Structural hole-based approach to control public opinion in a social network. Engineering Applications of Artificial Intelligence, 93, 103690. https://doi.org/10.1016/j.engappai.2020.103690

    Article  Google Scholar 

  • Guia, M., Silva, R., & Bernardino, J. (2019). Comparison of naïve bayes, support vector machine, decision trees and random forest on sentiment analysis. In: International Conference on Knowledge Discovery and Information Retrieval, pp. 525–531. SciTePress. https://doi.org/10.5220/0008364105250531

  • He, R., Lee, W. S., Ng, H. T., et al. (2018). Exploiting document knowledge for aspect-level sentiment classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, vol. 2, pp. 579–585. Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-2092

  • Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9, 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  • Huang, B., & Carley, K. (2018). Parameterized convolutional neural networks for aspect level sentiment classification. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1091–1096. Association for Computational Linguistics. https://doi.org/10.18653/v1/D18-1136

  • Huang, B., Ou, Y., & Carley, K. M. (2018). Aspect level sentiment classification with attention-over-attention neural networks. In: Social, Cultural, and Behavioral Modeling, vol. 10899, pp. 197–206. Springer. https://doi.org/10.1007/978-3-319-93372-6_22

  • Imani, M., & Noferesti, S. (2022). Aspect extraction and classification for sentiment analysis in drug reviews. Journal of Intelligent Information Systems, 59, 613–633. https://doi.org/10.1007/s10844-022-00712-w

    Article  Google Scholar 

  • Jaiswal, S., Mehta, A., & Nandi, G. (2018). Investigation on the effect of l1 an l2 regularization on image features extracted using restricted boltzmann machine. In: 2018 Second International Conference on Intelligent Computing and Control Systems (ICICCS), pp. 1548–1553. IEEE. https://doi.org/10.1109/ICCONS.2018.8663071

  • Karimi, A., Rossi, L., & Prati, A. (2021). Adversarial training for aspect-based sentiment analysis with bert. In: 2020 25th International Conference on Pattern Recognition, pp. 8797–8803. IEEE. https://doi.org/10.1109/ICPR48806.2021.9412167

  • Karimi, A., Rossi, L., & Prati, A. (2021). Improving bert performance for aspect-based sentiment analysis. In: Proceedings of the 4th International Conference on Natural Language and Speech Processing, pp. 39–46. Association for Computational Linguistics. https://aclanthology.org/2021.icnlsp-1.5

  • Kim Phung, T., An Te, N., & Thi Thu Ha, T. (2021). A machine learning approach for opinion mining online customer reviews. In: 2021 21st ACIS International Winter Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD-Winter), pp. 243–246. IEEE. https://doi.org/10.1109/SNPDWinter52325.2021.00059

  • Lan, Z., Chen, M., Goodman, S., et al. (2020). Albert: A lite bert for self-supervised learning of language representations. In: 8th International Conference on Learning Representations, pp. 1–17. OpenReview.net. https://doi.org/10.48550/arXiv.1909.11942

  • Li, X., Bing, L., Lam, W., et al. (2018). Transformation networks for target-oriented sentiment classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 946–956. Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-1087

  • Liu, P., Qiu, X., & Huang, X. (2016). Recurrent neural network for text classification with multi-task learning. In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, pp. 2873–2879. AAAI Press. https://doi.org/10.48550/arXiv.1605.05101

  • Liu, J., & Zhang, Y. (2017). Attention modeling for targeted sentiment. In: Conference of the European Chapter of the Association for Computational Linguistics, vol. 2, pp. 572–577. Association for Computational Linguistics. https://doi.org/10.18653/v1/E17-2091

  • Loshchilov, I., & Hutter, F. (2017). Fixing weight decay regularization in adam. Computing Research Repository abs/1711.05101, 1–13 (2017). https://doi.org/10.48550/arXiv.1711.05101

  • Ma, D., Li, S., Zhang, X., et al. (2017). Interactive attention networks for aspect-level sentiment classification. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pp. 4068–4074. AAAI Press. https://doi.org/10.24963/ijcai.2017/568

  • Ma, D., Wang, G., Han, C., et al. (2018). A memristive neural network model with associative memory for modeling affections. IEEE Access, 6, 61614–61622. https://doi.org/10.1109/ACCESS.2018.2875433

    Article  Google Scholar 

  • Meng, W., Wei, Y., Liu, P., et al. (2019). Aspect based sentiment analysis with feature enhanced attention cnn-bilstm. IEEE Access, 7, 167240–167249. https://doi.org/10.1109/ACCESS.2019.2952888

    Article  Google Scholar 

  • Mewada, A., & Dewang, R. K. (2022). Sa-asba: a hybrid model for aspect-based sentiment analysis using synthetic attention in pre-trained language bert model with extreme gradient boosting. The Journal of Supercomputing, 79, 5516–5551. https://doi.org/10.1007/s11227-022-04881-x

    Article  Google Scholar 

  • Morente-Molinera, J. A., Kou, G., Peng, Y., et al. (2018). Analysing discussions in social networks using group decision making methods and sentiment analysis. Information Sciences, 447, 157–168. https://doi.org/10.1016/j.ins.2018.03.020

    Article  Google Scholar 

  • Pontiki, M., Galanis, D., Papageorgiou, H., et al. (2016). Semeval-2016 task 5: Aspect based sentiment analysis. In: International Workshop on Semantic Evaluation, pp. 19–30. Association for Computational Linguistics. https://doi.org/10.18653/v1/S16-1002

  • Pontiki, M., Galanis, D., Pavlopoulos, J., et al. (2014). Semeval-2014 task 4: Aspect based sentiment analysis. In: Proceedings of the 8th International Workshop on Semantic Evaluation, pp. 27–35. Association for Computational Linguistics. https://doi.org/10.3115/v1/S14-2004

  • Rajpurkar, P., Jia, R., & Liang, P. (2018). Know what you don’t know: Unanswerable questions for squad. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, vol. 2, pp. 784–789. Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-2124

  • Ramaswamy, S. L., & Chinnappan, J. (2022). Recognet-lstm+cnn: a hybrid network with attention mechanism for aspect categorization and sentiment classification. Journal of Intelligent Information Systems, 58, 379–404. https://doi.org/10.1007/s10844-021-00692-3

    Article  Google Scholar 

  • Santucci, V., Forti, L., Santarelli, F., et al. (2020). Learning to classify text complexity for the italian language using support vector machines. In: Computational Science and Its Applications - ICCSA 2020, vol. 12250, pp. 367–376. Springer. https://doi.org/10.1007/978-3-030-58802-1_27

  • Singh, L. G., & Singh, S. R. (2021). Empirical study of sentiment analysis tools and techniques on societal topics. Journal of Intelligent Information Systems, 56, 379–407. https://doi.org/10.1007/s10844-020-00616-7

    Article  Google Scholar 

  • Song, Y., Wang, J., Jiang, T., et al. (2019). Targeted sentiment classification with attentional encoder network. In: Artificial Neural Networks and Machine Learning – ICANN 2019: Text and Time Series, vol. 11730, pp. 93–103. Springer. https://doi.org/10.1007/978-3-030-30490-4_9

  • Song, W., Wen, Z., Xiao, Z., et al. (2021). Semantics perception and refinement network for aspect-based sentiment analysis. Knowledge-Based Systems, 214, 106755. https://doi.org/10.1016/j.knosys.2021.106755

    Article  Google Scholar 

  • Sukhbaatar, S., Szlam, A., Weston, J., et al. (2015). End-to-end memory networks. In: Proceedings of the 28th International Conference on Neural Information Processing Systems, vol. 2, pp. 2440–2448. MIT Press. https://doi.org/10.48550/arXiv.1503.08895

  • Sun, C., Huang, L., & Qiu, X. (2019). Utilizing bert for aspect-based sentiment analysis via constructing auxiliary sentence. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 380–385. Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1035

  • Suzuki, M., Itoh, N., Nagano, T., et al. (2019). Improvements to n-gram language model using text generated from neural language model. In: ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7245–7249. IEEE. https://doi.org/10.1109/ICASSP.2019.8683481

  • Tang, D., Qin, B., Feng, X., et al. (2016). Effective lstms for target-dependent sentiment classification. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 3298–3307. The COLING 2016 Organizing Committee. https://doi.org/10.48550/arXiv.1512.01100

  • Tang, D., Qin, B., & Liu, T. (2016). Aspect level sentiment classification with deep memory network. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 214–224. Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1021

  • Tran, N. K., & Niedereée, C. (2018). Multihop attention networks for question answer matching. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 325–334. Association for Computing Machinery. https://doi.org/10.1145/3209978.3210009

  • Wang, Y., Huang, M., Zhu, X., et al. (2016). Attention-based lstm for aspect-level sentiment classification. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 606–615. Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1058

  • Wang, Q., Xu, J., Chen, H., et al. (2017). Two improved continuous bag-of-word models. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 2851–2856. IEEE. https://doi.org/10.1109/IJCNN.2017.7966208

  • Weston, J., Chopra, S., & Bordes, A. (2015). Memory networks. In: 3rd International Conference on Learning Representations, pp. 1–15. Ithaca, NY: arXiv.org. https://doi.org/10.48550/arXiv.1410.3916

  • Wu, C., Xiong, Q., Yang, Z., et al. (2021). Residual attention and other aspects module for aspect-based sentiment analysis. Neurocomputing, 435. https://doi.org/10.1016/j.neucom.2021.01.019

  • Xu, H., Liu, B., Shu, L., et al. (2019). Bert post-training for review reading comprehension and aspect-based sentiment analysis. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 2324–2335. Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1242

  • Xu, H., Liu, B., Shu, L., et al. (2019). Bert post-training for review reading comprehension and aspect-based sentiment analysis. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 2324–2335. Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1242

  • Xue, W., & Li, T. (2018). Aspect based sentiment analysis with gated convolutional networks. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 2514–2523. Association for Computational Linguistics (2018). https://doi.org/10.18653/v1/P18-1234

  • Zhang, Y., Ding, Q., Zhu, Z., et al. (2022). Enhancing aspect and opinion terms semantic relation for aspect sentiment triplet extraction, 59, 523–542. https://doi.org/10.1007/s10844-022-00710-y

    Article  Google Scholar 

  • Zhang, Z., Han, X., Liu, Z., et al. (2019). Ernie: Enhanced language representation with informative entities. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1441–1451. Association for Computational Linguistics. https://doi.org/10.18653/v1/P19-1139

  • Zhang, L., Wang, S., & Liu, B. (2018). Deep learning for sentiment analysis: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8, 1253. https://doi.org/10.1002/widm.1253

    Article  Google Scholar 

  • Zhou, J., Huang, X., Hu, Q., et al. (2020). Sk-gcn: Modeling syntax and knowledge via graph convolutional network for aspect-level sentiment classification. Knowledge-Based Systems, 205, 106292. https://doi.org/10.1016/j.knosys.2020.106292

    Article  Google Scholar 

  • Zhu, P., & Qian, T. (2018). Enhanced aspect level sentiment classification with auxiliary memory. In: Proceedings of the 27th International Conference on Computational Linguistics, pp. 1077–1087. Association for Computational Linguistics. https://aclanthology.org/C18-1092/

  • Zhu, L., Zhu, X., Guo, J., et al. (2022). Exploring rich structure information for aspect-based sentiment classification. Journal of Intelligent Information Systems, 60, 1–21. https://doi.org/10.1007/s10844-022-00729-1

    Article  Google Scholar 

Download references

Acknowledgements

Not Applicable.

Funding

This paper was supported by the Open Fund of Hubei Key Laboratory of Intelligent Geo-Information Processing (ZRIGIP-201801).

Author information

Authors and Affiliations

  1. School of Computer Science, China University of Geosciences, Wuhan, 430074, China

    Jiamei Wang, Wei Wu & Jiansi Ren

  2. Hubei Key Laboratory of Intelligent Geo-Information Processing, China University of Geosciences, Wuhan, 430074, China

    Jiansi Ren

Authors
  1. Jiamei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiansi Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Not Applicable.

Corresponding author

Correspondence to Jiansi Ren.

Ethics declarations

Ethical approval

Not Applicable.

Competing interests

The authors affirm that they do not have any competing interests. The founders were not involved in the design of the study, data collecting, analysis, or interpretation, article preparation, or decision-making.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, J., Wu, W. & Ren, J. BERT-PG: a two-branch associative feature gated filtering network for aspect sentiment classification. J Intell Inf Syst 60, 709–730 (2023). https://doi.org/10.1007/s10844-023-00785-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10844-023-00785-1

Keywords

Search

Navigation