Skip to main content
SpringerLink
Log in

SpSAN: Sparse self-attentive network-based aspect-aware model for sentiment analysis

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Consumer reviews for services and products are an essential performance measure for organizations on their offerings. They are also necessary for forthcoming consumers to understand previous consumer experiences. This experimental work carried out using consumer reviews gathered from an online review platform. This work evaluates consumer sentiment associated with qualitative content, quantitative ratings, and cultural aspect to predict Consumer Recommendation Decisions (CRDs). Moreover, we extract service aspects from online reviews and fuse them with the word sequences before feeding them into the model, which helps incorporate aspect representation and position information in context with the sentences. Additionally, this study proposed a Sparse Self-Attention Network (SpSAN) model to predict CRDs. Proposed SpSAN improves the fine-tuning performance of the Bidirectional Encoder Representations from Transformers (BERT) model by introducing sparsity into the self-attention procedure. Specifically, this work integrates sparsity into the self-attention mechanism by changing the softmax function with a controllable sparse transformation at the time of fine-tuning with BERT. It empowers us to understand sparse attention distribution with a more intelligible representation of the complete input data. Experimental results and their analysis describes the importance of the proposed SpSAN model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  • Abbasi-Moud Z, Vahdat-Nejad H, Sadri J (2021) Tourism recommendation system based on semantic clustering and sentiment analysis. Expert Syst Appl 167:114324

    Google Scholar 

  • Adhikari A, Ram A, Tang R, Lin J (2019) Docbert: Bert for document classification. CoRR, abs/1904.08398

  • Afzaal M, Usman M, Fong A (2019) Tourism mobile app with aspect-based sentiment classification framework for tourist reviews. IEEE Trans Consum Electron 65(2):233–242

    Google Scholar 

  • Ajorlou A, Jadbabaie A, Kakhbod A (2018) Dynamic pricing in social networks: The word-of-mouth effect. Manag Sci 64(2):971–979

    Google Scholar 

  • Anderson EW, Fornell C, Lehmann DR (1994) Customer satisfaction, market share, and profitability: findings from sweden. J Mark 58(3):53–66

    Google Scholar 

  • Arnould EJ, Thompson CJ (2005) Consumer culture theory (cct): Twenty years of research. J Consum Res 31(4):868–882

    Google Scholar 

  • Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: Bengio Y, LeCun, Y, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

  • Bansal B, Srivastava S (2019) Hybrid attribute based sentiment classification of online reviews for consumer intelligence. Appl Intell 49(1):137–149

    Google Scholar 

  • Bigné E, Zanfardini M, Andreu L (2020) How online reviews of destination responsibility influence tourists’ evaluations: an exploratory study of mountain tourism. J Sustain Tour 28(5):686–704

  • Cakici AC, Akgunduz Y, Yildirim O (2019) The impact of perceived price justice and satisfaction on loyalty: the mediating effect of revisit intention. Tour Rev 74(3):443–462

    Google Scholar 

  • Casillo M, Colace F, De Santo M, Lombardi M, Santaniello D (2021) A chatbot for training employees in industry 4.0. In:Research and Innovation Forum 2020: Disruptive Technologies in Times of Change, pp 397–409. Springer International Publishing

  • Chang K-C (2015) How travel agency reputation creates recommendation behavior. Ind Manag Data Syst

  • Chang Y-C, Ku C-H, Chen C-H (2019) Social media analytics: extracting and visualizing hilton hotel ratings and reviews from tripadvisor. Int J Inf Manag 48:263–279

    Google Scholar 

  • Chatterjee S (2019) Explaining customer ratings and recommendations by combining qualitative and quantitative user generated contents. Decis Support Syst 119:14–22

    Google Scholar 

  • Chatterjee S, Mandal P (2020) Traveler preferences from online reviews: Role of travel goals, class and culture. Tour Manag 80:104108

    Google Scholar 

  • Chen C, Zhuo R, Ren J (2019) Gated recurrent neural network with sentimental relations for sentiment classification. Inf Sci 502:268–278

    Google Scholar 

  • Cheng J, Dong L, Lapata M (2016) Long short-term memory-networks for machine reading. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas. Association for Computational Linguistics

  • Cheung CM, Lee MK (2012) What drives consumers to spread electronic word of mouth in online consumer-opinion platforms. Decis Support Syst 53(1):218–225

    Google Scholar 

  • Chevalier JA, Mayzlin D (2006) The effect of word of mouth on sales: Online book reviews. J Mark Res 43(3):345–354

    Google Scholar 

  • Clarizia F, Colace F, Pascale F, Lombardi M, Santaniello D (2019) Sentiment analysis in social networks: A methodology based on the latent dirichlet allocation approach. In: 11th Conference of the European Society for Fuzzy Logic and Technology (EUSFLAT 2019), pages 241–248. Atlantis Press

  • Colace F, De Santo M, Lombardi M, Santaniello D (2019) Chars: a cultural heritage adaptive recommender system. In: Proceedings of the 1st ACM International Workshop on Technology Enablers and Innovative Applications for Smart Cities and Communities, pp 58–61

  • Crotts JC, Erdmann R (2000) Does national culture influence consumers’ evaluation of travel services? a test of hofstede’s model of cross-cultural differences. Manag Serv Qual Int J

  • Devlin J, Chang M-W, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics

  • Duan W, Gu B, Whinston AB (2008) The dynamics of online word-of-mouth and product sales-an empirical investigation of the movie industry. J Retail 84(2):233–242

    Google Scholar 

  • Duda RO, Hart PE, Stork DG (2012) Pattern classification. John Wiley & Sons, Hoboken

    MATH  Google Scholar 

  • Farooq M, Raju V, Khalil-Ur-Rehman F, Ahmed Q, Ali M (2019) Investigating relationship between net promoter score and company performance: a longitudinal study. Glob J Emerg Sci 1(1):1–17

    Google Scholar 

  • Feldman R, Sanger J et al (2007) The text mining handbook: advanced approaches in analyzing unstructured data. Cambridge University Press, Cambridge

  • Grewal R, Chandrashekaran M, Citrin AV (2010) Customer satisfaction heterogeneity and shareholder value. J Mark Res 47(4):612–626

    Google Scholar 

  • Guerreiro J, Rita P (2019) How to predict explicit recommendations in online reviews using text mining and sentiment analysis. J Hospital Tour Manag, pp 269–272

  • Hallowell R (1996) The relationships of customer satisfaction, customer loyalty, and profitability: an empirical study. Int J Serv Ind Manag 7(4):27–42

    Google Scholar 

  • Hasan M, Uddin A, Alim A, Azad R, Ali B (2020) The effects of corporate image, service quality, consumer satisfaction, and perceived value on consumer loyalty: A field experience. Int Rev Bus Res Pap 16:160–176

    Google Scholar 

  • He R, Lee WS, Ng HT, Dahlmeier D (2017) An unsupervised neural attention model for aspect extraction. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 388–397, Vancouver, Canada. Association for Computational Linguistics

  • Hennig-Thurau T, Gwinner KP, Walsh G, Gremler DD (2004) Electronic word-of-mouth via consumer-opinion platforms: what motivates consumers to articulate themselves on the internet? J Interact Market 18(1):38–52

    Google Scholar 

  • Hofstede G (2001) Culture’s consequences: Comparing values, behaviors, institutions and organizations across nations. Sage publications

  • Hong H, Xu D, Wang GA, Fan W (2017) Understanding the determinants of online review helpfulness: a meta-analytic investigation. Decis Support Syst 102:1–11

    Google Scholar 

  • Hotho A, Nürnberger A, Paaß G (2005) A brief survey of text mining. In: Ldv Forum, volume 20, pp 19–62. Citeseer

  • Howard J, Ruder S (2018) Universal language model fine-tuning for text classification. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp 328–339, Melbourne, Australia. Association for Computational Linguistics

  • Huang M, Xie H, Rao Y, Feng J, Wang FL (2020) Sentiment strength detection with a context-dependent lexicon-based convolutional neural network. Inf Sci 520:389–399

    Google Scholar 

  • Jain PK, Pamula R, Ansari S, Sharma D, Maddala L (2019) Airline recommendation prediction using customer generated feedback data. In: 2019 4th International Conference on Information Systems and Computer Networks (ISCON), pages 376–379. IEEE

  • Jain PK, Pamula R, Srivastava G (2021) A systematic literature review on machine learning applications for consumer sentiment analysis using online reviews. Comput Sci Rev 41:100413

    Google Scholar 

  • Jain PK, Saravanan V, Pamula R (2021b) A hybrid cnn-lstm: a deep learning approach for consumer sentiment analysis using qualitative user-generated contents. Trans Asian Low-Resour Lang Inf Process 20(5):1–15

    Google Scholar 

  • Khorsand R, Rafiee M, Kayvanfar V (2020) Insights into tripadvisor’s online reviews: The case of tehran’s hotels. Tourism Management Perspectives 34:100673

  • Kim SS, Prideaux B (2003) A cross-cultural study of airline passengers. Ann Tour Res

  • Kingma DP, Ba J (2015) Adam: A method for stochastic optimization. In: Bengio Y, LeCun Y, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

  • Koch J, Gerdt S-O, Schewe G (2020) Determinants of sustainable behavior of firms and the consequences for customer satisfaction in hospitality. Int J Hosp Manag 89:102515

    Google Scholar 

  • Kohavi R et al (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. Ijcai, vol 14. Montreal, Canada, pp 1137–1145

  • Kong L, Li C, Ge J, Zhang F, Feng Y, Li Z, Luo B (2020) Leveraging multiple features for document sentiment classification. Inf Sci 518:39–55

    Google Scholar 

  • Laha A, Chemmengath SA, Agrawal P, Khapra M, Sankaranarayanan K, Ramaswamy HG (2018) On controllable sparse alternatives to softmax. In: Bengio S, Wallach H, Larochelle H, Grauman K, Cesa-Bianchi N, Garnett R (eds) Advances in Neural Information Processing Systems 31. Curran Associates Inc, pp 6422–6432

  • Lee J, Benjamin S, Childs M (2020) Unpacking the emotions behind tripadvisor travel reviews: The case study of gatlinburg, tennessee. Int J Hospital Tour Adm, pp 1–18

  • Li X, Wu C, Mai F (2019) The effect of online reviews on product sales: a joint sentiment-topic analysis. Inf Manag 56(2):172–184

    Google Scholar 

  • Liu X, He P, Chen W, Gao J (2019) Multi-task deep neural networks for natural language understanding. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp 4487–4496, Florence, Italy. Association for Computational Linguistics

  • Luong T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp 1412–1421, Lisbon, Portugal. Association for Computational Linguistics

  • Malaviy C, Ferreira P, Martins AFT (2018). Sparse and constrained attention for neural machine translation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia. Association for Computational Linguistics

  • Martensen A, Gronholdt L, Kristensen K (2000) The drivers of customer satisfaction and loyalty: cross-industry findings from denmark. Total Qual Manag 11(4–6):544–553

    Google Scholar 

  • Martins AFT, Astudillo RF (2016). From softmax to sparsemax: A sparse model of attention and multi-label classification. In: Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML’16, page 1614–1623. JMLR.org

  • Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Burges CJC, Bottou L, Welling M, Ghahramani Z, Weinberger KQ (eds) Advances in Neural Information Processing Systems 26. Curran Associates Inc, pp 3111–3119

  • Mitchell TM, et al. (1997) Machine learning. 1997. Burr Ridge, IL: McGraw Hill, 45(37):870–877

  • Mouwen A (2015) Drivers of customer satisfaction with public transport services. Transp Res Part Policy Pract 78:1–20

    Google Scholar 

  • Naseem U, Razzak I, Musial K, Imran M (2020) Transformer based deep intelligent contextual embedding for twitter sentiment analysis. Futur Gener Comput Syst 113:58–69

    Google Scholar 

  • Niculae V, Blondel M (2017) A regularized framework for sparse and structured neural attention. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in Neural Information Processing Systems 30. Curran Associates Inc, pp 3338–3348

  • Oh H (1999) Service quality, customer satisfaction, and customer value: a holistic perspective. Int J Hosp Manag 18(1):67–82

    Google Scholar 

  • Ozdemir S, Zhang S, Gupta S, Bebek G (2020) The effects of trust and peer influence on corporate brand-consumer relationships and consumer loyalty. J Bus Res 117:791–805

    Google Scholar 

  • Parikh A, Täckström O, Das D, Uszkoreit J (2016). A decomposable attention model for natural language inference. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp 2249–2255, Austin, Texas. Association for Computational Linguistics

  • Peters M, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L(2018) Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics

  • Popescu A-M, Etzioni O (2007) Extracting product features and opinions from reviews. Natural language processing and text mining. Springer, NewYork, pp 9–28

    Google Scholar 

  • Punel A, Hassan LAH, Ermagun A (2019) Variations in airline passenger expectation of service quality across the globe. Tour Manag 75:491–508

    Google Scholar 

  • Quamer W, Jain PK, Rai A, Saravanan V, Pamula R, Kumar C (2021) Sacnn: Self-attentive convolutional neural network model for natural language inference. Trans Asian Low-Resourc Lang Inf Process 20(3):1–16

    Google Scholar 

  • Radford A, Narasimhan K, Salimans T, Sutskever I (2018) Improving language understanding by generative pre-training

  • Reisinger Y, Crotts JC (2010) Applying hofstede’s national culture measures in tourism research: Illuminating issues of divergence and convergence. J Travel Res 49(2):153–164

    Google Scholar 

  • Saha GC, et al (2009) Service quality, satisfaction, and behavioural intentions. Manag Serv Qual Int J

  • Salton G, Buckley C (1988) Term-weighting approaches in automatic text retrieval. Inf Process Manag 24(5):513–523

    Google Scholar 

  • Sezgen E, Mason KJ, Mayer R (2019) Voice of airline passenger: a text mining approach to understand customer satisfaction. J Air Transp Manag 77:65–74

    Google Scholar 

  • Siering M, Deokar AV, Janze C (2018) Disentangling consumer recommendations: explaining and predicting airline recommendations based on online reviews. Decis Support Syst 107:52–63

    Google Scholar 

  • Söderlund M (1998) Customer satisfaction and its consequences on customer behaviour revisited. Int J Serv Ind Manag 9(2):169–188

    Google Scholar 

  • Steenkamp J-BE (2019) Global versus local consumer culture: theory, measurement, and future research directions. J Int Mark 27(1):1–19

    Google Scholar 

  • Taecharungroj V, Mathayomchan B (2019) Analysing tripadvisor reviews of tourist attractions in phuket, thailand. Tour Manag 75:550–568

    Google Scholar 

  • Tanford S (2013) The impact of tier level on attitudinal and behavioral loyalty of hotel reward program members. Int J Hosp Manag 34:285–294

    Google Scholar 

  • Tang F, Fu L, Yao B, Xu W (2019) Aspect based fine-grained sentiment analysis for online reviews. Inf Sci 488:190–204

    Google Scholar 

  • Taylor SA, Baker TL (1994) An assessment of the relationship between service quality and customer satisfaction in the formation of consumers’ purchase intentions. J Retail 70(2):163–178

    Google Scholar 

  • Thomas M-J, Wirtz BW, Weyerer JC (2019) Determinants of online review credibility and its impact on consumers’purchase intention. J Electron Commer Res 20(1):1–20

    Google Scholar 

  • Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, u., and Polosukhin, I. (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in Neural Information Processing Systems 30. Curran Associates Inc, pp 5998–6008

  • Vijayarani S, Ilamathi MJ, Nithya M (2015) Preprocessing techniques for text mining-an overview. Int J Comput Sci Commun Netw 5(1):7–16

    Google Scholar 

  • Wang W, Yang N, Wei F, Chang B, Zhou M (2017) Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189–198, Vancouver, Canada. Association for Computational Linguistics

  • Weston J, Chopra S, Bordes A (2015) Memory networks. In: Bengio, Y, LeCun, Y, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings

  • Yang W, Xie Y, Lin A, Li X, Tan L, Xiong K, Li M, Lin J (2019) End-to-end open-domain question answering with BERTserini. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp 72–77, Minneapolis, Minnesota. Association for Computational Linguistics

  • Zhang M, Palade V, Wang Y, Ji Z (2021) Attention-based word embeddings using artificial bee colony algorithm for aspect-level sentiment classification. Inf Sci 545:713–738

    Google Scholar 

  • Zhao Y, Wang L, Tang H, Zhang Y (2020) Electronic word-of-mouth and consumer purchase intentions in social e-commerce. Electron Commer Res Appl 41:100980

    Google Scholar 

  • Zhou J, Chen Q, Huang JX, Hu QV, He L (2020) Position-aware hierarchical transfer model for aspect-level sentiment classification. Inf Sci 513:1–16

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Indian Institute of Technology (Indian School of Mines), Dhanbad, JH, 826004, India

    Praphula Kumar Jain, Waris Quamer & Rajendra Pamula

  2. Vassar College, Poughkeepsie, NY, USA

    Vijayalakshmi Saravanan

Authors
  1. Praphula Kumar Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Waris Quamer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rajendra Pamula
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vijayalakshmi Saravanan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Praphula Kumar Jain.

Ethics declarations

Conflict of interest

All authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jain, P.K., Quamer, W., Pamula, R. et al. SpSAN: Sparse self-attentive network-based aspect-aware model for sentiment analysis. J Ambient Intell Human Comput 14, 3091–3108 (2023). https://doi.org/10.1007/s12652-021-03436-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-021-03436-x

Keywords

Search

Navigation