Abstract
When people buy products online, they primarily base their decisions on the recommendations of others given in online reviews. The current work analyzed these online reviews by sentiment analysis and used the extracted sentiments as features to predict the product ratings by several machine learning algorithms. These predictions were disentangled by various methods of explainable AI (XAI) to understand whether the model showed any bias during prediction.
Study 1 benchmarked these algorithms (knn, support vector machines, random forests, gradient boosting machines, XGBoost) and identified random forests and XGBoost as best algorithms for predicting the product ratings. In Study 2, the analysis of global feature importance identified the sentiment joy and the emotional valence negative as most predictive features. Two XAI visualization methods, local feature attributions and partial dependency plots, revealed several incorrect prediction mechanisms on the instance-level. Performing the benchmarking as classification, Study 3 identified a high no-information rate of 64.4% that indicated high class imbalance as underlying reason for the identified problems.
In conclusion, good performance by machine learning algorithms must be taken with caution because the dataset, as encountered in this work, could be biased towards certain predictions. This work demonstrates how XAI methods reveal such prediction bias.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Carvalho, D.V., Pereira, E.M., Cardoso, J.S.: Machine learning interpretability: a survey on methods and metrics. Electronics 8, 1–34 (2019). https://doi.org/10.3390/electronics8080832
Rudin, C.: Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat. Mach. Intell. 1, 206–215 (2019). https://doi.org/10.1038/s42256-019-0048-x
Mudambi, S.M., Schuff, D.: What makes a helpful online review? A study of customer reviews on amazon.com. MIS Q. Manag. Inf. Syst. 34, 185–200 (2010). https://doi.org/10.2307/20721420
Bansal, B., Srivastava, S.: On predicting elections with hybrid topic based sentiment analysis of tweets. Procedia Comput. Sci. 135, 346–353 (2018). https://doi.org/10.1016/j.procs.2018.08.183
Akhtar, N., Zubair, N., Kumar, A., Ahmad, T.: Aspect based sentiment oriented summarization of hotel reviews. Procedia Comput. Sci. 115, 563–571 (2017). https://doi.org/10.1016/j.procs.2017.09.115
Hasan, A., Moin, S., Karim, A., Shamshirband, S.: Machine learning-based sentiment analysis for twitter accounts. Math. Comput. Appl. 23, 11 (2018). https://doi.org/10.3390/mca23010011
Yu, Y.: Aspect-based Sentiment Analysis on Hotel Reviews. Arxiv Preprint 10 (2016)
de Kok, S., Punt, L., van den Puttelaar, R., Ranta, K., Schouten, K., Frasincar, F.: Review-level aspect-based sentiment analysis using an ontology. In: Proceedings of the ACM Symposium on Applied Computing, pp. 315–322 (2018). https://doi.org/10.1145/3167132.3167163
Ekman, P.: Are there basic emotions? Psychol. Rev. 99, 550–553 (1992). https://doi.org/10.4081/jear.2011.169
Vytal, K., Hamann, S.: Neuroimaging support for discrete neural correlates of basic emotions: a voxel-based meta-analysis. J. Cogn. Neurosci. 22, 2864–2885 (2010). https://doi.org/10.1162/jocn.2009.21366
Saarimäki, H., et al.: Discrete Neural Signatures of Basic Emotions. Cereb. Cortex 26, 2563–2573 (2016). https://doi.org/10.1093/cercor/bhv086
Rucker, D.D., Petty, R.E.: Emotion specificity and consumer behavior: anger, sadness, and preference for activity. Motiv. Emot. 28, 3–21 (2004). https://doi.org/10.1023/B:MOEM.0000027275.95071.82
Delplanque, S., Lavoie, M.E., Hot, P., Silvert, L., Sequeira, H.: Modulation of cognitive processing by emotional valence studied through event-related potentials in humans. Neurosci. Lett. 356, 1–4 (2004). https://doi.org/10.1016/j.neulet.2003.10.014
Bohrn, I.C., Altmann, U., Lubrich, O., Menninghaus, W., Jacobs, A.M.: When we like what we know - a parametric fMRI analysis of beauty and familiarity. Brain Lang. 124, 1–8 (2013). https://doi.org/10.1016/j.bandl.2012.10.003
Kuhn, M.: Caret: classification and regression training (2018)
Karatzoglou, A., Smola, A., Hornik, K., Zeileis, A.: kernlab – an S4 package for kernel methods in R. J. Stat. Softw. 11, 1–20 (2004)
Therneau, T., Atkinson, B.: rpart: recursive partitioning and regression trees (2019)
Liaw, A., Wiener, M.: Classification and regression by randomForest. R News 2, 18–22 (2002)
Greenwell, B., Boehmke, B., Cunningham, J., Developers, G.B.M.: gbm: Generalized Boosted Regression Models (2019)
Chen, T., Guestrin, C.: XGBoost: A Scalable Tree Boosting System. Arxiv (2016)
Doshi-Velez, F., Kim, B.: Towards a rigorous science of interpretable machine learning. 1–13 (2017)
Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29(5), 1189–1232 (2001). https://doi.org/10.2307/2699986
Acknowledgment
This research was supported by the Yonsei University Faculty Research Fund of 2019-22-0199.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
So, C. (2020). What Emotions Make One or Five Stars? Understanding Ratings of Online Product Reviews by Sentiment Analysis and XAI. In: Degen, H., Reinerman-Jones, L. (eds) Artificial Intelligence in HCI. HCII 2020. Lecture Notes in Computer Science(), vol 12217. Springer, Cham. https://doi.org/10.1007/978-3-030-50334-5_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-50334-5_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-50333-8
Online ISBN: 978-3-030-50334-5
eBook Packages: Computer ScienceComputer Science (R0)