Abstract
This paper presents our efforts to develop a scale for measuring students’ learning experiences with knowledge-based chatbots in massive open online courses (MOOCs) through three studies. In Study 1, we conducted a qualitative synthesis of the current literature and analyzed students’ open-ended responses regarding their experiences with a knowledge-based chatbot. Consequently, we identified eight salient domains (i.e., social presence, teaching presence, cognitive presence, self-regulation, co-regulation, perceived ease of use, behavioral intention, and enjoyment), resulting in the creation of 53 items. In Study 2, we selected 30 items that received more than 80% agreement from five experts. Finally, in Study 3, we reported the findings of exploratory and confirmatory factor analyses of the final scale based on student responses (N = 237) and presented 22 items across five domains (i.e., social presence, teaching and cognitive presence, self-regulation, perceived ease of use, and behavioral intention). This research contributes to the current literature by providing an instrument to measure students’ learning experiences with knowledge-based chatbots in MOOCs, which is presently unavailable. The scale developed in this study could be employed for further research aiming to systematically develop knowledge-based chatbots and investigate the relationships between salient factors influencing students’ learning experiences in MOOCs.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The quantitative data set analyzed in this study is available from the corresponding author upon request.
References
Arbaugh, J. B., Bangert, A., & Cleveland-Innes, M. (2010). Subject matter effects and the community of Inquiry (CoI) framework: An exploratory study. The Internet and Higher Education, 13(1), 37–44. https://doi.org/10.1016/j.iheduc.2009.10.006.
Bazelais, P., Doleck, T., & Lemay, D. J. (2018). Investigating the predictive power of TAM: A case study of CEGEP students’ intentions to use online learning technologies. Education and Information Technologies, 23(1), 93–111. https://doi.org/10.1007/s10639-017-9587-0.
Befus, M. (2016). Conducting a multivocal thematic synthesis on an extensive body of literature. Canadian Journal of Learning and Technology/La Revue Canadienne de l’apprentissage et de La Technologie, 42(2). https://www.learntechlib.org/d/178102.
Bentler, P. M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. https://doi.org/10.1037/0033-2909.107.2.238.
Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), 588–606. https://doi.org/10.1037/0033-2909.88.3.588.
Billings, D. M. (2000). A Framework for assessing outcomes and practices in web-based courses in nursing. Journal of Nursing Education, 39(2), 60–67. https://doi.org/10.3928/0148-4834-20000201-07.
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2018.00149
Bott, N., Wexler, S., Drury, L., Pollak, C., Wang, V., Scher, K., & Narducci, S. (2019). A protocol-driven, bedside digital conversational agent to support nurse teams and mitigate risks of hospitalization in older adults: Case control pre-post study. Journal of Medical Internet Research, 21(10), e13440. https://doi.org/10.2196/13440.
Chang, C. Y., Kuo, S. Y., & Hwang, G. H. (2022). Chatbot-facilitated nursing education: Incorporating a knowledge-based chatbot system into a nursing training program. Educational Technology and Society, 25(1). https://www.jstor.org/stable/48647027.
Cleveland-Innes, M., & Campbell, P. (2012). Emotional presence, learning, and the online learning environment. The International Review of Research in Open and Distributed Learning, 13(4), 269–292. https://doi.org/10.19173/irrodl.v13i4.1234.
Cohen, P., West, S. G., & Aiken, L. S. (2014). Applied multiple regression/correlation analysis for the behavioral sciences. Psychology Press.
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008.
Desjardins, C. D., & Bulut, O. (2018). Handbook of educational measurement and psychometrics using R. CRC Press.
Deveci Topal, A., Eren, D., C., & Kolburan Geçer, A. (2021). Chatbot application in a 5th grade science course. Education and Information Technologies, 26(5), 6241–6265. https://doi.org/10.1007/s10639-021-10627-8.
DeVellis, R. F., & Thorpe, C. T. (2021). Scale development: Theory and applications. SAGE Publications.
Garrison, D. R. (2017). E-learning in the 21st century: A community of inquiry framework for research and practice (3rd ed.). Routledge. https://doi.org/10.4324/9781315667263
Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Lawrence Erlbaum Associates Inc.
Groves, R., Floyd, F., Couper, M., Lepkowski, J., Singer, E., & Tourangeau, R. (2011). Survey methodology. Cham: Wiley.
Gupta, S., & Yu, C. (2022). Supporting inclusive learning using chatbots? A chatbot-led interview study. Journal of Information Systems Education, 33(1), 98–108. https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1966&context=jise.
Han, S., & Lee, M. K. (2022). FAQ chatbot and inclusive learning in massive open online courses. Computers & Education. https://doi.org/10.1016/j.compedu.2021.104395
Harrison, P. J., Saba, F., Seeman, B. J., Molise, G., Behm, R., & Williams, M. D. (1991). Development of a distance education assessment instrument. Educational Technology Research and Development, 39(4), 65–77. https://doi.org/10.1007/BF02296572.
Hassani, K., Nahvi, A., & Ahmadi, A. (2013). Architectural design and implementation of intelligent embodied conversational agents using fuzzy knowledge base. Journal of Intelligent & Fuzzy Systems, 25(3), 811–823.
Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., & Mctear, M. (2019). Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces? Proceedings of the 31st European Conference on Cognitive Ergonomics, pp. 207–214. https://doi.org/10.1145/3335082.3335094.
Hooper, D., Coughlan, J., & Mullen, M. R. (2008). Structural equation modeling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, 6(1), 53–60.
Hsu, I., & Yu, J. (2022). A medical chatbot using machine learning and natural language understanding. Multimedia Tools and Applications, 81(17), 23777–23799. https://doi.org/10.1007/s11042-022-12820-4.
Hsu, M., Chan, T., & Yu, C. (2023). Termbot: A chatbot-based crossword game for gamified medical terminology learning. International Journal of Environmental Research and Public Health. https://doi.org/10.3390/ijerph20054185
Kline, P. (1994). An easy guide to factor analysis. Routledge. https://doi.org/10.4324/9781315788135
Kohnke, L. (2022). A qualitative exploration of student perspectives of chatbot use during emergency remote teaching. International Journal of Mobile Learning and Organisation, 16(4), 475–488. https://doi.org/10.1504/IJMLO.2022.125966.
Kohnke, L. (2023). L2 learners’ perceptions of a chatbot as a potential independent language learning tool. International Journal of Mobile Learning and Organisation, 17(1–2), 214–226. https://doi.org/10.1504/IJMLO.2023.128339.
Kovanović, V., Joksimović, S., Poquet, O., Hennis, T., Čukić, I., de Vries, P., Hatala, M., Dawson, S., Siemens, G., & Gašević, D. (2018). Exploring communities of inquiry in massive open online courses. Computers & Education, 119, 44–58. https://doi.org/10.1016/j.compedu.2017.11.010.
Kozak, M. (2009). What is strong correlation? Teaching Statistics, 31(3), 85–86. https://doi.org/10.1111/j.1467-9639.2009.00387.x.
Kuhail, M. A., Alturki, N., Alramlawi, S., & Alhejori, K. (2022). Interacting with educational chatbots: A systematic review. Education and Information Technologies. https://doi.org/10.1007/s10639-022-11177-3.
Li, L., Johnson, J., Aarhus, W., & Shah, D. (2022). Key factors in MOOC pedagogy based on NLP sentiment analysis of learner reviews: What makes a hit. Computers & Education. https://doi.org/10.1016/j.compedu.2021.104354.
Lin, M. P., & Chang, D. (2020). Enhancing post-secondary writers’ writing skills with a chatbot: A mixed-method classroom study. Journal of Educational Technology & Society, 23(1), 78–92.
Lin, Y., & Ye, J. (2023). Development of an educational chatbot system for enhancing students’ biology learning performance. Journal of Internet Technology, 24(2), 275–281. https://doi.org/10.53106/160792642023032402006.
Liu, I. F., Chen, M. C., Sun, Y. S., Wible, D., & Kuo, C. H. (2010). Extending the TAM model to explore the factors that affect intention to use an online learning community. Computers & Education, 54(2), 600–610.
Maheshwari, G. (2021). Factors affecting students’ intentions to undertake online learning: An empirical study in Vietnam. Education and Information Technologies, 26(6), 6629–6649. https://doi.org/10.1007/s10639-021-10465-8.
Malik, R., Sharma, A., Trivedi, S., & Mishra, R. (2021). Adoption of chatbots for learning among university students: Role of perceived convenience and enhanced performance. International Journal of Emerging Technologies in Learning, 16(18), 200–212. https://www.learntechlib.org/p/220124/.
Meyers, L. S., Gamst, G., & Guarino, A. J. (2017). Applied multivariate research: Design and interpretation. Sage publications.
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & PRISMA Group. (2009). Reprint–preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Physical Therapy, 89(9), 873–880.
Morgado, F. F. R., Meireles, J. F. F., Neves, C. M., Amaral, A. C. S., & Ferreira, M. E. C. (2017). Scale development: Ten main limitations and recommendations to improve future research practices. Reflexão e Crítica. https://doi.org/10.1186/s41155-016-0057-1
Naji, K. K., Du, X., Tarlochan, F., Ebead, U., Hasan, M. A., & Al-Ali, A. K. (2020). Engineering students’ readiness to transition to emergency online learning in response to COVID-19: Case of Qatar. EURASIA Journal of Mathematics, Science and Technology Education, 16(10). https://eric.ed.gov/?id=EJ1284842
Neff, G., & Nagy, P. (2016). Talking to bots: Symbiotic agency and the case of Tay. International Journal of Communication, 19328036, 10, 4915–4931.
Nunnally, J. C. (1967). Psychometric theory. McGraw-Hill.
Nuruzzaman, M., & Hussain, O. K. (2018). A survey on chatbot implementation in customer service industry through deep neural networks. 2018 IEEE 15th International Conference on E-Business Engineering (ICEBE), pp. 54–61. https://doi.org/10.1109/ICEBE.2018.00019.
Park, S., Choi, J., Lee, S., Oh, C., Kim, C., La, S., Lee, J., & Suh, B. (2019). Designing a chatbot for a brief motivational interview on stress management: Qualitative case study. Journal of Medical Internet Research, 21(4), e12231. https://doi.org/10.2196/12231.
Picciano, A. G. (2021). Theories and frameworks for online education: Seeking an integrated model. A guide to administering Distance Learning (pp. 79–103). Brill.
Prokes, C., & Housel, J. (2021). Community college student perceptions of remote learning shifts due to COVID-19. TechTrends, 65(4), 576–588. https://doi.org/10.1007/s11528-021-00587-8.
Samrose, S., & Hoque, E. (2022). MIA: Motivational interviewing agent for improving conversational skills in remote group discussions. Proceedings of the ACM on Human-Computer Interaction, 6(GROUP), 1–24. https://doi.org/10.1145/3492864.
Shah, V., Murthy, S., Warriem, J., Sahasrabudhe, S., Banerjee, G., & Iyer, S. (2022). Learner-centric MOOC model: A pedagogical design model towards active learner participation and higher completion rates. Educational Technology Research and Development, 70(1), 263–288. https://doi.org/10.1007/s11423-022-10081-4.
Shao, C. (2020). An empirical study on the identification of driving factors of satisfaction with online learning based on TAM. 5th International Conference on Economics, Management, Law and Education (EMLE 2019), pp. 1067–1073. https://doi.org/10.2991/aebmr.k.191225.205.
Sheppard, B. H., Hartwick, J., & Warshaw, P. R. (1988). The theory of reasoned action: A meta-analysis of past research with recommendations for modifications and future research. Journal of Consumer Research, 15(3), 325–343. https://doi.org/10.1086/209170.
Spiliotopoulos, D., Makri, E., Vassilakis, C., & Margaris, D. (2020). Multimodal interaction: Correlates of learners’ metacognitive skill training negotiation experience. Information. https://doi.org/10.3390/info11080381
Stenbom, S., Hrastinski, S., & Cleveland-Innes, M. (2016). Emotional presence in a relationship of inquiry: The case of one-to-one online math coaching. Online Learning, 20(1), 41–56. https://files.eric.ed.gov/fulltext/EJ1096376.pdf.
Ursachi, G., Horodnic, I. A., & Zait, A. (2015). How reliable are measurement scales? External factors with indirect influence on reliability estimators. Procedia Economics and Finance, 20, 679–686. https://doi.org/10.1016/S2212-5671(15)00123-9.
Vázquez-Cano, E., Mengual-Andrés, S., & López-Meneses, E. (2021). Chatbot to improve learning punctuation in spanish and to enhance open and flexible learning environments. International Journal of Educational Technology in Higher Education. https://doi.org/10.1186/s41239-021-00269-8
Venkatesh, V., & Davis, F. D. (1996). A model of the antecedents of perceived ease of use: Development and test. Decision Sciences, 27(3), 451–481.
Yang, H., & Yoo, Y. (2004). It’s all about attitude: Revisiting the technology acceptance model. Decision Support Systems, 38(1), 19–31. https://doi.org/10.1016/S0167-9236(03)00062-9.
Yildiz Durak, H. (2023). Conversational agent-based guidance: Examining the effect of chatbot usage frequency and satisfaction on visual design self-efficacy, engagement, satisfaction, and learner autonomy. Education and Information Technologies, 28(1), 471–488. https://doi.org/10.1007/s10639-022-11149-7.
Zhang, S., Zhao, J., & Tan, W. (2008). Extending TAM for online learning systems: An intrinsic motivation perspective. Tsinghua Science and Technology, 13(3), 312–317. https://doi.org/10.1016/S1007-0214(08)70050-6.
Acknowledgements
We extend our sincere gratitude to Drs. Jason Harron, Martin Hlosta, Amy Ogan, Zilong Pan, and Wenting Ellen Zou for their invaluable expertise and contribution to the content validity evaluation in Study 2. Our gratitude extends to Dr. Henry May for the valuable suggestion to use IRT for scale enhancement in the future. Their insightful feedback and guidance have significantly enriched our research.
Author information
Authors and Affiliations
Contributions
SH: Conceptualization, methodology, software, validation, formal analysis, investigation, data curation, writing (original and editing), visualization. XH: conceptualization, methodology, validation, investigation, resources, supervision. YC and PS: conceptualization, data curation, validation, investigation, original draft writing. ML: conceptualization, methodology, resources, supervision.
Corresponding author
Ethics declarations
Competing interests
The authors report no potential conflicts of interest.
Ethical approval
The Institutional Review Board at the University of Texas at Austin reviewed and approved this study before the research.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Han, S., Hamilton, X., Cai, Y. et al. Knowledge-based chatbots: a scale measuring students’ learning experiences in massive open online courses. Education Tech Research Dev 71, 2431–2456 (2023). https://doi.org/10.1007/s11423-023-10280-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11423-023-10280-7