Abstract
Massive open online courses (MOOCs) are potentially participated in by very many students from different parts of the world, which means that learning analytics is especially challenging. In this framework, predicting students’ performance is a key issue, but the high level of heterogeneity affects understanding and measurement of the causal links between performance and its drivers, including motivation, attitude to learning, and engagement, with different models recommended for the formulation of appropriate policies. Using data for the FedericaX EdX MOOC platform (Federica WebLearning Centre at the University of Naples Federico II), we exploit a consolidated composite-based path model to relate performance with engagement and learning. The model addresses heterogeneity by analysing gender, age, country of origin, and course design differences as they affect performance. Results reveal subgroups of students requiring different learning strategies to enhance final performance. Our main findings were that differences in performance depended mainly on learning for male students taking instructor-paced courses, and on engagement for older students (> 32 years) taking self-paced courses.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Students from Africa and Oceania were merged in a single group labelled “Others” due to the small number of cases. This classification was justified by similar average performance values for students from those two geographical regions.
References
Allione, G., Stein, R.M.: Mass attrition: an analysis of Drop Out from Principles of Microeconomics MOOC. J. Econ. Educ. 47(2), 174–186 (2016)
Azevedo, R.: Defining and measuring engagement and learning in science: conceptual, theoretical, methodological, and analytical issues. Educ. Psychol. 50(1), 84–94 (2015)
Bradley, W., Henseler, J.: Modeling Reflective Higher-order Constructs using Three Approaches with Pls Path Modeling: A Monte Carlo Comparison (2007) https://hdl.handle.net/2066/160877
Carannante, M., Davino, C., Vistocco, D.: Modelling students’ performance in moocs: a multivariate approach. Stud. High. Educ. (2020). https://doi.org/10.1080/03075079.2020.1723526
Cheah, J.H., Amaro, S., Roldán, J.L.: Multigroup analysis of more than two groups in PLS-SEM: a review, illustration, and recommendations. J. Bus. Res. (2023). https://doi.org/10.1016/j.jbusres.2022.113539
Conijn, R., Van den Beemt, A., Cuijpers, P.: Predicting student performance in a blended MOOC. J. Comput. Assist. 34(5), 1–14 (2018). https://doi.org/10.1111/jcal.12270
Dabbagh, N.: The online learner: characteristics and pedagogical implications. Contemp. Issues Technol. Teacher Educ. 7(3), 217–226 (2007)
de Barba, P.G., Kennedy, G.E., Ainley, M.D.: The role of students’ motivation and participation in predicting performance in a MOOC. J. Comput. Assist. 32, 218–231 (2016). https://doi.org/10.1111/jcal.12130
Deng, R., Benckendorff, P., Gannaway, D.: Linking learner factors, teaching context, and engagement patterns with MOOC learning outcomes. J. Comput. Assist. 36(5), 688–708 (2020). https://doi.org/10.1111/jcal.12437
Dweck, C.S.: Motivational processes affecting learning. Am. Psychol. 41(10), 1040–1048 (1986). https://doi.org/10.1037/0003-066X.41.10.1040
Endedijk, M.D., Brekelmans, M., Sleegers, P., et al.: Measuring students’ self-regulated learning in professional education: bridging the gap between event and aptitude measurements. Qual. Quant. 50, 2141–2164 (2016). https://doi.org/10.1007/s11135-015-0255-4
Esposito Vinzi, V., Chin, W.W., Henseler, J., Wang, H.: Handbook of Partial Least Squares: Concepts, Methods and Applications. Springer, Berlin Heidelberg (2010)
Fianu, E., Blewett, C., Ampong, G.O.A., Ofori, K.S.: Factors affecting MOOC usage by students in selected Ghanaian Universities. Sci. Educ. 8(2), 70 (2018)
Fenollar, P., Román, S., Cuestas, P.J.: University students’ academic performance: an integrative conceptual framework and empirical analysis. J. Educ. Psychol. 77(4), 873–891 (2007). https://doi.org/10.1348/000709907X189118
Gameel, B.G., Wilkins, K.G.: When it Comes to MOOCs, where you are from Makes a Difference. Comput. Educ.. Educ. 136, 49–60 (2019). https://doi.org/10.1016/j.compedu.2019.02.014
Gefen, D., Rigdon, E.E., Straub, D.: Editor’s comments: an update and extension to SEM guidelines for administrative and social science research. MIS Quart. iii–xiv (2011). https://doi.org/10.2307/23044042
Ghasemy, M., Teeroovengadum, V., Becker, J.M., Ringle, C.M.: This fast car can move faster: a review of PLS-SEM application in higher education research. High. Educ. 80(6), 1121–1152 (2020). https://doi.org/10.1007/s10734-020-00534-1
Guàrdia, J., Freixa, M., Peró, M., et al.: Factors related to the academic performance of students in the statistics course in psychology. Qual. Quant. 40, 661–674 (2006). https://doi.org/10.1007/s11135-005-2072-7
Goopio, J., Cheung, C.: The MOOC dropout phenomenon and retention strategies. J. Teach. Travel Tour. (2020). https://doi.org/10.1080/15313220.2020.1809050
Green, S.B.: How many subjects does it take to do a regression analysis. Multivar. Behav. Res. 26(3), 499–510 (1991). https://doi.org/10.1207/s15327906mbr2603_7
Guo, P.J., Reinecke, K.: Demographic Differences in how Students Navigate through MOOCs. In: Proceedings of the first ACM conference on Learning@ scale conference (pp. 21–30) (2014). https://doi.org/10.1145/2556325.2566247
Hair Jr, J.F., Sarstedt, M., Ringle, C., Gudergan, S.P.: Advanced Issues in Partial Least Squares Structural Equation Modeling. Sage publications. (2017)
Hair Jr, J.F., Hult, G.T.M., Ringle, C., Sarstedt, M.: A Primer on Partial Least Squares Structural Equation Modeling (PLS-SEM). Sage publications. (2016)
Hadwin, A.F., Nesbit, J.C., Jamieson-Noel, D., Code, J., Winne, P.H.: Examining trace data to explore self-regulated learning. Metacogn. Learn.. Learn. 2, 107–124 (2007)
Henseler, J., Ringle, C., Sarstedt, M.: Testing measurement invariance of composites using partial least squares. Int. Mark. Rev. 33(3), 405–431 (2016)
Henseler, J., Ringle, C., Sinkovics, R.R.: The use of partial least squares path modeling in international marketing. Adv. Int. Mark. 20, 273–319 (2009). https://doi.org/10.1108/S1474-7979(2009)0000020014
Henderikx, M.A., Kreijns, K., Kalz, M.: Refining success and dropout in massive open online courses based on the intention–behavior gap. Distance Educ. 38(3), 353–368 (2017). https://doi.org/10.1080/01587919.2017.1369006
Hintze, J.L., Nelson, R.D.: Violin plots: a box plot-density trace synergism. Am. Stat. 52(2), 181–184 (1998). https://doi.org/10.1080/00031305.1998.10480559
Hood, N., Littlejohn, A., Milligan, C.: Context counts: how learners’ contexts influence learning in a MOOC. Comput. Educ.. Educ. 91, 83–91 (2015). https://doi.org/10.1016/j.iheduc.2015.12.003
Kahu, E.R.: Framing student engagement in higher education. Stud. High. Educ. 38(5), 758–773 (2013)
Kizilcec, R.F., Pérez-Sanagustín, M., Maldonado, J.J.: Self-regulated learning strategies predict learner behavior and goal attainment in massive open online courses. Comput. Educ.. Educ. 104, 18–33 (2017). https://doi.org/10.1016/j.compedu.2016.10.001
Krasodomska, J., Godawska,: E-learning in accounting education: the influence of students’ characteristics on their engagement and performance. J. Account. Educ. (2020). https://doi.org/10.1080/09639284.2020.1867874
Lamberti, G., Banet Aluja, T., Sanchez, G.: The pathmox approach for pls path modeling: discovering which constructs differentiate segments. Appl. Stoch. Models. Bus. Ind. 33(6), 674–689 (2017). https://doi.org/10.1002/asmb.2270
Lamberti, G., Banet Aluja, T., Sanchez, G.: The pathmox approach for PLS path modeling segmentation. Appl. Stoch. Models. Bus. Ind. 32(4), 453–468 (2016). https://doi.org/10.1002/asmb.2168
Lamberti, G.: Hybrid multigroup partial least squares structural equation modelling: an application to bank employee satisfaction and loyalty. Qual. Quant. (2021). https://doi.org/10.1007/s11135-021-01096-9
Lamberti, G., Aluja Banet, T., Rialp Criado, J.: Work climate drivers and employee heterogeneity. Int. J. Hum. Resour. Manag.Resour. Manag. 33(3), 472–504 (2022). https://doi.org/10.1080/09585192.2020.1711798
Lan, M., Hew, K.F.: Examining learning engagement in MOOCs: a self-determination theoretical perspective using mixed method. Int. J. Educ. Technol. (2020). https://doi.org/10.1186/s41239-020-0179-5
Lebart, L., Morineau, A., Fenelon, J.P.: Traitement des Donnees Statistiques. Dunod, Paris (1979)
Lee, J.: Racial and ethnic achievement gap trends: Reversing the Progress Toward Equity? Educ. Res. 31(1), 3–12 (2002). https://doi.org/10.3102/0013189X031001003
Li, K.: MOOC learners’ demographics, self-regulated learning strategy, perceived learning and satisfaction: a structural equation modeling approach. Comput. Educ.. Educ. 132, 16–30 (2019). https://doi.org/10.1016/j.compedu.2019.01.003
Lietaert, S., Roorda, D., Laevers, F., Verschueren, K., De Fraine, B.: The gender gap in student engagement: the role of teachers’ autonomy support, structure, and involvement. Br. J. Educ. Psychol. 85(4), 498–518 (2015). https://doi.org/10.1111/bjep.12095
Lim, D.H., Morris, M.L., Yoon, S.W.: Combined effect of instructional and learner variables on course outcomes within an online learning environment. Interact. Learn. Environ. 5(3), 255–269 (2006)
Lim, J.M.: Predicting successful completion using student delay indicators in undergraduate self-paced online courses. Distance Educ. 37(3), 317–332 (2016). https://doi.org/10.1080/01587919.2016.1233050
Lu, H.P., Lin, J.C.C., Hsiao, K.L., Cheng, L.T.: Information sharing behaviour on blogs in Taiwan: effects of interactivities and gender differences. Inf. 36(3), 401–416 (2010)
Maya-Jariego, I., Holgado, D., González-Tinoco, E., Castaño-Muñoz, J., Punie, Y.: Typology of motivation and learning intentions of users in MOOCs: the MOOCKNOWLEDGE study. Educ. Technol. Res. Dev. 68(1), 203–224 (2020)
Moore, R.L., Wang, C.: Influence of learner motivational dispositions on MOOC completion. Comput. Educ.. Educ. 33(1), 121–134 (2021). https://doi.org/10.1007/s12528-020-09258-8
Ngah, A.H., Kamalrulzaman, N.I., Mohamad, M., et al.: Do science and social science differ? Multi-group analysis (MGA) of the willingness to continue online learning. Qual. Quant. (2022). https://doi.org/10.1007/s11135-022-01465-y
Ngah, A.H., Ramayah, T., Ali, M.H., Khan, M.I.: Halal transportation adoption among pharmaceuticals and comestics manufacturers. J. Islamic Mark. 11(6), 1619–1639 (2019). https://doi.org/10.1108/JIMA-10-2018-0193
Pellizzari, M., Billari, F.C.: The younger, the better? Age-related differences in academic performance at university. J. Popul. Econ.Popul. Econ. 25, 697–739 (2012). https://doi.org/10.1007/s00148-011-0379-3
Phan, T., McNeil, S.G., Robin, B.R.: Students’ patterns of engagement and course performance in a massive open online course. Comput. Educ.. Educ. 95, 36–44 (2016)
Rai, L., Yue, Z., Yang, T., Shadiev, R., Sun, N.: General Impact of MOOC Assessment Methods on Learner Engagement and Performance. In 2017 10th International Conference on Ubi-media Computing and Workshops (Ubi-Media) (pp. 1–4). IEEE. (2017)
Ramesh, A., Goldwasser, D., Huang, B., Iii, H.D., Getoor, L.: Learning Latent Engagement Patterns of Students in Online Courses. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pp. 1272–1278. AAAI Press. (2014)
Ramsay, S., Barker, M., Jones, E.: Academic adjustment and learning processes: a comparison of international and local students in first-year university. High. Educ. Res. 18(1), 129–144 (1999). https://doi.org/10.1080/0729436990180110
Rizvi, S., Rienties, B., Khoja, S.A.: The role of demographics in online learning; a decision tree based approach. Comput. Educ.. Educ. 137, 32–47 (2019). https://doi.org/10.1016/j.compedu.2019.04.001
Robertson, M., Line, M., Jones, S., Thomas, S.: International students, learning environments and perceptions: a case study using the Delphi technique. High. Educ. Res. 19(1), 89–102 (2000). https://doi.org/10.1080/07294360050020499
Reed, W.M., Oughton, J.M.: Computer experience and interval-based hypermedia navigation. J. Res. Technol. Educ. 30(1), 38–52 (1997). https://doi.org/10.1080/08886504.1997.10782212
Sarstedt, M., Hair, J.F., Ringle, C.M., Thiele, K.O., Gudergan, S.P.: Estimation issues with PLS and CBSEM: where the bias lies!. J. Bus. Res. 69(10), 3998–4010 (2016). https://doi.org/10.1016/j.jbusres.2016.06.00
Shao, Z., Chen, K.: Understanding individuals’ engagement and continuance intention of MOOCs: the effect of interactivity and the role of gender. Internet Res. (2020). https://doi.org/10.1108/INTR-10-2019-0416
Shafaei, A., Nejati, M., Quazi, A., Von der Heidt, T.: ‘When in Rome, do as the Romans do’Do international students’ acculturation attitudes impact their ethical academic conduct? High. Educ. 71(5), 651–666 (2016). https://doi.org/10.1007/s10734-015-9928-0
Song, L., Singleton, E.S., Hill, J.R., Koh, M.H.: Improving online learning: student perceptions of useful and challenging characteristics. Internet High. Educ. 7(1), 59–70 (2014). https://doi.org/10.1016/j.iheduc.2003.11.003
Sullivan, P.: Gender differences and the online classroom: male and female college students evaluate their experiences. Commun. Coll. J. Res. Pract. 25, 805–818 (2001). https://doi.org/10.1080/106689201753235930
Tani, M., Gheith, M.H., Papaluca, O.: Drivers of student engagement in higher education: a behavioral reasoning theory perspective. High. Educ. 82(3), 499–518 (2021)
Taplin, M., Jegede, O.: Gender differences in factors influencing achievement of distance education students. Open Learn. 16(2), 133–154 (2001). https://doi.org/10.1080/02680510120050307
Tezer, M., Yildiz, E.P., Uzunboylu, H.: Online authentic learning self-efficacy: a scale development. Qual. Quant. 52(Suppl 1), 639–649 (2018). https://doi.org/10.1007/s11135-017-0641-1
Timms, C., Fishman, T., Godineau, A., Granger, J., Sibanda, T.: Psychological engagement of university students: learning communities and family relationships. J. Appl. Res. High. Educ 10(3), 243–255 (2018). https://doi.org/10.1108/JARHE-09-2017-0107
Tuckman, B.W.: Relations of academic procrastination, rationalizations, and performance in a web course with deadlines. Psychol. Rep. 96(3), 1015–1021 (2005). https://doi.org/10.2466/pr0.96.3c.1015-1021
van Dinther, M., Dochy, F., Segers, M.: Factors affecting students’ self-efficacy in higher education. Educ. Res. Rev. 6(2), 95–108 (2011). https://doi.org/10.1016/j.edurev.2010.10.003
Veletsianos, G., Kimmons, R., Larsen, R., Rogers, J.: Temporal flexibility, gender, and online learning completion. Dist. Educ. 42(1), 22–36 (2021). https://doi.org/10.1080/01587919.2020.1869523
Vermunt, J.D.: Relations between student learning patterns and personal and contextual factors and academic performance. High. Educ. 49(3), 205–234 (2005)
Wang, M.T., Willett, J.B., Eccles, J.S.: The assessment of school engagement: examining dimensionality and measurement invariance by gender and race/ethnicity. J. Sch. Psychol. 49(4), 465–480 (2011). https://doi.org/10.1016/j.jsp.2011.04.001
Watson, W.R., Yu, J.H., Watson, S.L.: Perceived attitudinal learning in a self-paced versus fixed-schedule MOOC. Educ. Media Int. 55(2), 170–181 (2018). https://doi.org/10.1080/09523987.2018.1484044
Weiser, E.B.: Gender differences in internet use patterns and internet application preferences: a two-sample comparison. Cyberpsychology 3(2), 167–178 (2000)
Williams, K.M., Stafford, R.E., Corliss, S.B., Reilly, E.D.: Examining student characteristics, goals, and engagement in massive open online courses. Comput. Educ.. Educ. 126, 433–442 (2018). https://doi.org/10.1016/j.compedu.2018.08.014
Wold, H.: Partial least squares. In S. Kotz e N. Johnson (Eds.). Enc. of Stat. Scien. John Wiley & Sons (1985)
Xiong, Y., Li, H., Kornhaber, M.L., Suen, H.K., Pursel, B., Goins, D.D.: Examining the relations among student motivation, engagement, and retention in a MOOC: a structural equation modeling approach. Glob. Educ. Rev. 2(3), 23–33 (2015)
Yukselturk, E., Bulut, S.: Gender differences in self-regulated online learning environment. J. Educ. Techno. Soc. 12(3), 12–22 (2009)
You, J.W.: Identifying significant indicators using LMS data to predict course achievement in online learning. Internet High Educ. 29, 23–30 (2016). https://doi.org/10.1016/j.iheduc.2015.11.003
Zhao, C.M., Kuh, G.D., Carini, R.M.: A comparison of international student and American student engagement in effective educational practices. J. Higher Educ. 76(2), 209–231 (2005). https://doi.org/10.1080/00221546.2005.11778911
Funding
This work was supported by grant #870691-INVENT funded by the EU H2020 program.
Author information
Authors and Affiliations
Contributions
Authors contributed equally.
Corresponding author
Ethics declarations
Conflict of interest
The authors report there are no competing interests to declare.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Appendix
Appendix
Indicators are reported as violin plots (Hintze and Nelson 1998), as these simultaneously depict the full distribution and number of data considered. The height of each violin indicates the range of the detected values, while the width indicates the position of the peak. In the separate panels, the colours reflect the six subdimensions used to measure learning, engagement, and performance. Thus, indicators related to frequency-based actions, time-based actions, and interaction are coloured green, light blue, and red, respectively, while regularity, non-procrastination, and performance are coloured dark blue, yellow, and violet, respectively. All indicators are highly skewed, with long tails on the right of the distributions (Fig. 4).
Summary statistics for student learning, engagement, and performance indicators
Tables 6, 7, 8. Measurement model reliability. Crossloadings, bootstrap confidence intervals (CIs) calculated with 500 repetitions, composite reliability (CR), and average expected variance (AVE) results for the first-order constructs, as reported in Table 6. Results are acceptable, as, according to Esposito Vinzi et al. (2010), CR should be greater than 0.7, AVE should be greater than 0.5, and loadings are higher than 0.7 and significantly higher with respect to their own constructs. Loadings, bootstrap CIs, CR, and AVE results for the second-order constructs are reported in Table 7. Loading values for both constructs are lower than 0.7 (explained by the particular nature of the indicators used in the analysis) but are significant, CR is higher than 0.7 for both constructs, AVE is higher than 0.5 for engagement, and although lower than 0.5 for learning, is still close to the threshold. Given that CR is higher than 0.7, convergent validity can still be considered adequate (Fornell and Larcker, 1981). Results for the Fornell-Larcker matrix, reported in Table 8, indicate that discriminant validity is assured.
Tables 9, 10, 11, and 12 MICOM testing of the invariance measurement model; steps (2) and (3). Each table reports the observed score correlation (SC), the 5% confidence interval (CI), and the observed score difference in means (SDM) between the compared groups, and the log-ratio of score variances (LSV) for groups with their corresponding 95% CIs (obtained by group permutation). Compositional invariance is verified when the SC value falls within the CI, and full measurement invariance is verified when SDM and LSV values fall within the CI. Note that in case of Table 12 (course design), the SC is lower than the threshold, although, in this case, the observed deviation occurs in the third decimal. According to Lamberti et al. (2022), the correlation is not too low for MGA and so the compositional invariance of the constructs is globally accepted.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Cristina, D., Giuseppe, L. & Domenico, V. Assessing heterogeneity in MOOC student performance through composite-based path modelling. Qual Quant 58, 2453–2477 (2024). https://doi.org/10.1007/s11135-023-01760-2
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11135-023-01760-2