Abstract
In Massive Open Online Courses (MOOCs), learners can post both text comments and overall ratings regarding the courses. There is growing interest in assessing the consistency of online reviews and the determinants of learner satisfaction. This study analyses the disconfirmation effect between textual review topics and the determinants of learner satisfaction in MOOCs. The MOOCs are categorised under three disciplines - Social Science, Technical Science, and Humanities & Natural Science. A crawler was employed to collect the corpus, extracting 93,679 reviews of 5,214 online courses from a Chinese university MOOC platform (icourse163.org). Textual analytics was used in the topic extraction. The empirical results suggest a strong disconfirmation effect between textual reviews and the determinants of learner satisfaction, i.e., not all textual review topics affect the overall learner satisfaction. Compared with positive reviews, negative (and neutral) reviews have a stronger disconfirmation effect. Further, the antecedents of learner attention are course-discipline specific. The disconfirmation effect is course-discipline dependent, with the most prominent for Technical Science courses, and the least for Humanities & Natural Science courses. This study provides a framework to guide platform managers and course instructors in better course delivery and enhancing overall learner satisfaction.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The datasets analysed during the current study are available from the corresponding author on reasonable request.
References
Acheampong, F. A., Cehn, W., & Nunoo-Mensah, H. (2020). Text-based emotion detection: Advances, challenges, and opportunities. Engineering Reports, 2(7), e12189. https://doi.org/10.1002/eng2.12189
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T
Ashaari, M. A., Singh, K. S. D., Abbasi, G. A., Amran, A., & Liebana-Cabanillas, F. J. (2021). Big data analytics capability for improved performance of higher education institutions in the era of IR 4.0: A multi-analytical SEM & ANN perspective. Technological Forecasting and Social Change, 173, 121119. https://doi.org/10.1016/j.techfore.2021.121119
Bem, D. J. (1967). Self-perception: An alternative interpretation of cognitive dissonance phenomena. Psychological Review, 74(3), 183–200. https://doi.org/10.1037/h0024835
Bhattacherjee, A. (2001). Understanding information systems continuance: An expectation-confirmation model. MIS Quarterly, 25(3), 351–370. https://doi.org/10.2307/3250921
Boos, D. D., & Duan, S. (2021). Pairwise comparisons using ranks in the one-way model. The American Statistician, 75(4), 414–423. https://doi.org/10.1080/00031305.2020.1860819
Brauwers, G., & Frasincar, F. (2022). A survey on aspect-based sentiment classification. ACM Computing Surveys, 55(4), 1–37. https://doi.org/10.1145/3503044
Breslow, N. (1970). A generalized Kruskal-Wallis test for comparing k samples subject to unequal patterns of censorship. Biometrika, 57(3), 579–594. https://doi.org/10.1093/biomet/57.3.579
Cai, M., Luo, H., Meng, X., Cui, Y., & Wang, W. (2023). Network distribution and sentiment interaction: Information diffusion mechanisms between social bots and human users on social media. Information Processing & Management, 60(2), 103197. https://doi.org/10.1016/j.ipm.2022.103197
Conover, W. J., & Iman, R. L. (1981). Rank transformations as a bridge between parametric and nonparametric statistics. The American Statistician, 35(3), 124–129. https://doi.org/10.2307/2683975
Dai, H. M., Teo, T., & Rappa, N. A. (2020a). Understanding continuance intention among MOOC participants: The role of habit and MOOC performance. Computers in Human Behavior, 112, 106455. https://doi.org/10.1016/j.chb.2020.106455
Dai, H. M., Teo, T., Rappa, N. A., & Huang, F. (2020b). Explaining chinese university students’ continuance learning intention in the MOOC setting: A modified expectation confirmation model perspective. Computers & Education, 150, 103850. https://doi.org/10.1016/j.compedu.2020.103850
Dai, C. P., Ke, F., & Pan, Y. (2022). Narrative-supported math problem solving in digital game-based learning. Educational Technology Research and Development, 70(4), 1261–1281. https://doi.org/10.1007/s11423-022-10129-5
Deng, R., & Benckendorff, P. (2021). What are the key themes associated with the positive learning experience in MOOCs? An empirical investigation of learners’ ratings and reviews. International Journal of Educational Technology in Higher Education, 18(1), 1–28. https://doi.org/10.1186/s41239-021-00244-3
Ding, Y., & Zhao, T. (2020). Emotions, engagement, and self-perceived achievement in a small private online course. Journal of Computer Assisted Learning, 36(4), 449–457. https://doi.org/10.1111/jcal.12410
Doo, M. Y., Zhu, M., Bonk, C. J., & Tang, Y. (2020). The effects of openness, altruism and instructional self-efficacy on work engagement of MOOC instructors. British Journal of Educational Technology, 51(3), 743–760. https://doi.org/10.1111/bjet.12882
Edwards, O. V., & Taasoobshirazi, G. (2022). Social presence and teacher involvement: The link with expectancy, task value, and engagement. The Internet and Higher Education, 55, 100869. https://doi.org/10.1016/j.iheduc.2022.100869
Fan, J., Jiang, Y., Liu, Y., & Zhou, Y. (2021). Interpretable MOOC recommendation: A multi-attention network for personalized learning behavior analysis. Internet Research, 32(2), 588–605. https://doi.org/10.1108/INTR-08-2020-0477
Gašević, D., Greiff, S., & Shaffer, D. W. (2022). Towards strengthening links between learning analytics and assessment: Challenges and potentials of a promising new bond. Computers in Human Behavior, 134, 107304. https://doi.org/10.1016/j.chb.2022.107304
Geng, S., Niu, B., Feng, Y., & Huang, M. (2020). Understanding the focal points and sentiment of learners in MOOC reviews: A machine learning and SC-LIWC-based approach. British Journal of Educational Technology, 51(5), 1785–1803. https://doi.org/10.1111/bjet.12999
Guan, C., Mou, J., & Jiang, Z. (2020). Artificial intelligence innovation in education: A twenty-year data-driven historical analysis. International Journal of Innovation Studies, 4(4), 134–147. https://doi.org/10.1016/j.ijis.2020.09.001
Harwell, M. R. (1988). Choosing between parametric and nonparametric tests. Journal of Counseling & Development, 67(1), 35–38. https://doi.org/10.1002/j.1556-6676.1988.tb02007.x
Hew, K. F., Hu, X., Qiao, C., & Tang, Y. (2020). What predicts student satisfaction with MOOCs: A gradient boosting trees supervised machine learning and sentiment analysis approach. Computers & Education, 145, 103724. https://doi.org/10.1016/j.compedu.2019.103724
Ho, Y. C., Wu, J., & Tan, Y. (2017). Disconfirmation effect on online rating behavior: A structural model. Information Systems Research, 28(3), 626–642. https://doi.org/10.1287/isre.2017.0694
Huang, X., Huang, L., & Lajoie, S. P. (2022). Exploring teachers’ emotional experience in a TPACK development task. Educational Technology Research and Development, 70(4), 1283–1303. https://doi.org/10.1007/s11423-022-10135-7
Khan, A. (2023). Improved multi-lingual sentiment analysis and recognition using deep learning. Journal of Information Science, 01655515221137270. https://doi.org/10.1177/01655515221137270
Kim, D., Jung, E., Yoon, M., Chang, Y., Park, S., Kim, D., & Demir, F. (2021). Exploring the structural relationships between course design factors, learner commitment, self-directed learning, and intentions for further learning in a self-paced MOOC. Computers & Education, 166, 104171. https://doi.org/10.1016/j.compedu.2021.104171
Kovanović, V., Joksimović, S., Poquet, O., Hennis, T., de Vries, P., Hatala, M., Dawson, S., Siemens, G., & Gašević, D. (2019). Examining communities of inquiry in massive open online courses: The role of study strategies. The Internet and Higher Education, 40, 20–43. https://doi.org/10.1016/j.iheduc.2018.09.001
Li, M. Y., Makino, S., & Jiang, C. (2019). Does national sentiment affect foreign direct investment, and if so, how? Additional evidence. International Business Review, 28(5), 101586. https://doi.org/10.1016/j.ibusrev.2019.101586
Lilliefors, H. W. (1967). On the Kolmogorov-Smirnov test for normality with mean and variance unknown. Journal of the American Statistical Association, 62(318), 399–402. https://doi.org/10.2307/2283970
Liu, J., Yu, Y., Mehraliyev, F., Hu, S., & Chen, J. (2022). What affects the online ratings of restaurant consumers: A research perspective on text-mining big data analysis. International Journal of Contemporary Hospitality Management, 34(10), 3607–3633. https://doi.org/10.1108/IJCHM-06-2021-0749
Martin, F., & Bolliger, D. U. (2022). Developing an online learner satisfaction framework in higher education through a systematic review of research. International Journal of Educational Technology in Higher Education, 19(1), 50. https://doi.org/10.1186/s41239-022-00355-5
Mubarak, A. A., Ahmed, S. A., & Cao, H. (2021). MOOC-ASV: Analytical statistical visual model of learners’ interaction in videos of MOOC courses. Interactive Learning Environments, 1–16. https://doi.org/10.1080/10494820.2021.1916768
Nanda, G., Douglas, K. A., Waller, D. R., Merzdorf, H. E., & Goldwasser, D. (2021). Analyzing large collections of open-ended feedback from MOOC learners using LDA topic modeling and qualitative analysis. IEEE Transactions on Learning Technologies, 14(2), 146–160. https://doi.org/10.1109/TLT.2021.3064798
Nie, R. X., Tian, Z. P., Wang, J. Q., & Chin, K. S. (2020). Hotel selection driven by online textual reviews: Applying a semantic partitioned sentiment dictionary and evidence theory. International Journal of Hospitality Management, 88, 102495. https://doi.org/10.1016/j.ijhm.2020.102495
Oliver, R. L. (1980). A cognitive model of the antecedents and consequences of satisfaction decisions. Journal of Marketing Research, 17(4), 460–469. https://doi.org/10.1177/002224378001700405
Rabin, E., Kalman, Y. M., & Kalz, M. (2019). An empirical investigation of the antecedents of learner-centered outcome measures in MOOCs. International Journal of Educational Technology in Higher Education, 16(1), 1–20. https://doi.org/10.1186/s41239-019-0144-3
Rawat, S., Kumar, D., Kumar, P., & Khattri, C. (2021). A systematic analysis using classification machine learning algorithms to understand why learners drop out of MOOCs. Neural Computing and Applications, 33(21), 14823–14835. https://doi.org/10.1007/s00521-021-06122-3
Reparaz, C., Aznárez-Sanado, M., & Mendoza, G. (2020). Self-regulation of learning and MOOC retention. Computers in Human Behavior, 111, 106423. https://doi.org/10.1016/j.chb.2020.106423
Romero-Frías, E., Arquero, J. L., & del Barrio-García, S. (2020). Exploring how student motivation relates to acceptance and participation in MOOCs. Interactive Learning Environments, 1–17. https://doi.org/10.1080/10494820.2020.1799020
Shah, V., Murthy, S., Warriem, J., Sahasrabudhe, S., Banerjee, G., & Iyer, S. (2022). Learner-centric MOOC model: A pedagogical design model towards active learner participation and higher completion rates. Educational Technology Research and Development, 70(1), 263–288. https://doi.org/10.1007/s11423-022-10081-4
Shang, S., & Lyv, W. (2022). Understanding the impact of quality elements on MOOCs continuance intention. Education and Information Technologies, 27(8), 10949–10976. https://doi.org/10.1007/s10639-022-11063-y
Shao, Z. (2018). Examining the impact mechanism of social psychological motivations on individuals’ continuance intention of MOOCs: The moderating effect of gender. Internet Research, 28(1), 232–250. https://doi.org/10.1108/IntR-11-2016-0335
Slevitch, L., & Oh, H. (2010). Asymmetric relationship between attribute performance and customer satisfaction: A new perspective. International Journal of Hospitality Management, 29(4), 559–569. https://doi.org/10.1016/j.ijhm.2009.09.004
Song, Y., Wang, R., Fernandez, J., & Li, D. (2021). Investigating sense of place of the Las Vegas Strip using online reviews and machine learning approaches. Landscape and Urban Planning, 205, 103956. https://doi.org/10.1016/j.landurbplan.2020.103956
Sun, Y., Ni, L., Zhao, Y., Shen, X. L., & Wang, N. (2019). Understanding students’ engagement in MOOCs: An integration of self-determination theory and theory of relationship quality. British Journal of Educational Technology, 50(6), 3156–3174. https://doi.org/10.1111/bjet.12724
Suwanaposee, P., Gutwin, C., & Cockburn, A. (2022). The influence of audio effects and attention on the perceived duration of interaction. International Journal of Human-Computer Studies, 159, 102756. https://doi.org/10.1016/j.ijhcs.2021.102756
Syed, S., & Weber, C. T. (2018). Using machine learning to uncover latent research topics in fishery models. Reviews in Fisheries Science & Aquaculture, 26(3), 319–336. https://doi.org/10.1080/23308249.2017.1416331
Taboada, M., Brooke, J., Tofiloski, M., Voll, K., & Stede, M. (2011). Lexicon-based methods for sentiment analysis. Computational Linguistics, 37(2), 267–307. https://doi.org/10.1162/COLI_a_00049
Tang, H. (2021). Teaching teachers to use technology through massive open online course: Perspectives of interaction equivalency. Computers & Education, 174, 104307. https://doi.org/10.1016/j.compedu.2021.104307
Tang, H., Arslan, O., Xing, W., & Kamali-Arslantas, T. (2022a). Exploring collaborative problem solving in virtual laboratories: A perspective of socially shared metacognition. Journal of Computing in Higher Education, 1–24. https://doi.org/10.1007/s12528-022-09318-1
Tang, H., Dai, M., Yang, S., Du, X., Hung, J. L., & Li, H. (2022b). Using multimodal analytics to systemically investigate online collaborative problem-solving. Distance Education, 43(2), 1–28. https://doi.org/10.1080/01587919.2022.2064824
Tseng, T. H., Lin, S., Wang, Y. S., & Liu, H. X. (2022). Investigating teachers’ adoption of MOOCs: The perspective of UTAUT2. Interactive Learning Environments, 30(4), 635–650. https://doi.org/10.1080/10494820.2019.1674888
Wang, W., Guo, L., He, L., & Wu, Y. J. (2019a). Effects of social-interactive engagement on the dropout ratio in online learning: Insights from MOOC. Behaviour & Information Technology, 38(6), 621–636. https://doi.org/10.1080/0144929X.2018.1549595
Wang, W., Guo, L., & Sun, R. (2019b). Rational herd behavior in online learning: Insights from MOOC. Computers in Human Behavior, 92, 660–669. https://doi.org/10.1016/j.chb.2017.10.009
Wang, W., Guo, L., & Wu, Y. J. (2022a). The merits of a sentiment analysis of antecedent comments for the prediction of online fundraising outcomes. Technological Forecasting and Social Change, 174, 121070. https://doi.org/10.1016/j.techfore.2021.121070
Wang, W., Zhao, Y., Wu, Y. J., & Goh, M. (2022b). Interaction strategies in online learning: Insights from text analytics on iMOOC. Education and Information Technologies, 1–28. https://doi.org/10.1007/s10639-022-11270-7
Wei, X., Saab, N., & Admiraal, W. (2021). Assessment of cognitive, behavioral, and affective learning outcomes in massive open online courses: A systematic literature review. Computers & Education, 163, 104097. https://doi.org/10.1016/j.compedu.2020.104097
Williams, K. M., Stafford, R. E., Corliss, S. B., & Reilly, E. D. (2018). Examining student characteristics, goals, and engagement in massive open online courses. Computers & Education, 126, 433–442. https://doi.org/10.1016/j.compedu.2018.08.014
Wu, B., & Wang, Y. (2022). Formation mechanism of popular courses on MOOC platforms: A configurational approach. Computers & Education, 191, 104629. https://doi.org/10.1016/j.compedu.2022.104629
Xing, W. (2022). Does the early bird catch the worm? A large-scale examination of the effects of early participation in online learning. Distance Education, 43(3), 466–481. https://doi.org/10.1080/01587919.2022.2088476
Xing, W., Tang, H., & Pei, B. (2019). Beyond positive and negative emotions: Looking into the role of achievement emotions in discussion forums of MOOCs. The Internet and Higher Education, 43, 100690. https://doi.org/10.1016/j.iheduc.2019.100690
Xu, X. (2020). Examining an asymmetric effect between online customer reviews emphasis and overall satisfaction determinants. Journal of Business Research, 106, 196–210. https://doi.org/10.1016/j.jbusres.2018.07.022
Yang, B., Tang, H., Hao, L., & Rose, J. R. (2022). Untangling chaos in discussion forums: A temporal analysis of topic-relevant forum posts in MOOCs. Computers & Education, 178, 104402. https://doi.org/10.1016/j.compedu.2021.104402
Zhu, Y., Xu, S., Wang, W., Zhang, L., Liu, D., Liu, Z., & Xu, Y. (2022). The impact of online and offline learning motivation on learning performance: The mediating role of positive academic emotion. Education and Information Technologies, 27(7), 8921–8938. https://doi.org/10.1007/s10639-022-10961-5
Zuo, M., Hu, Y., Luo, H., Ouyang, H., & Zhang, Y. (2022). K-12 students’ online learning motivation in China: An integrated model based on community of inquiry and technology acceptance theory. Education and Information Technologies, 27(4), 4599–4620. https://doi.org/10.1007/s10639-021-10791-x
Acknowledgements
This work is partially supported by the National Nature Science Foundation of China Grant (72072062), Natural Science Foundation of Fujian Province (2020J01782), and Ministry of Science & Technology, Taiwan (MOST 109-2511-H-003049-MY3).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
None.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Wang, W., Liu, H., Wu, Y.J. et al. Disconfirmation effect on online reviews and learner satisfaction determinants in MOOCs. Educ Inf Technol 28, 15497–15521 (2023). https://doi.org/10.1007/s10639-023-11824-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10639-023-11824-3