Abstract
While machine learning (ML) has been extensively used in Massive Open Online Courses (MOOCs) to predict whether learners are at risk of dropping-out or failing, very few work has investigated the bias or possible unfairness of the predictions generated by these models. This is however important, because MOOCs typically engage very diverse audiences worldwide, and it is unsure whether the existing ML models will generate fair predictions to all learners. In this paper, we explore the fairness of ML models meant to predict course completion in a MOOC mostly offered in Europe an Africa. To do so, we leverage and compare ABROCA and MADD, two fairness metrics that have been proposed specifically in education. Our results show that some ML models are more likely to generate unfair predictions than others. Even in the fairest models, we found biases in their predictions related to how the learners’ enrolled as well as their country, gender, age and job status. These biases are particularly detrimental to African learners, which is a key finding as they are an understudied population in AI fairness analysis in education.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Algeria, Benin, Burkina-Faso, Burundi, Cameroon, DR Congo, Ivory Coast, Djibouti, Gabon, Guinea, Guinea-Bissau, Madagascar, Mali, Morocco, Mauritania, Mozambique, Niger, Central African Republic, Republic of the Congo, Rwanda, Senegal, Tchad, Togo, Tunisia.
- 2.
- 3.
An alternative approach is to compute the features solely for the current week, thus “forgetting” the previous ones, however in our case, we found that both approaches yield similar results.
- 4.
The grid search space and default values are documented in our Github repository linked above.
- 5.
References
Baker, R.S., Hawn, A.: Algorithmic bias in education. Int. J. Artif. Intell. Educ. 1052–1092 (2021)
Bridgeman, B., Trapani, C., Attali, Y.: Considering fairness and validity in evaluating automated scoring. In: Proceedings of the Annual meeting of the National Council on Measurement in Education, San Diego, CA (2009)
Bridgeman, B., Trapani, C., Attali, Y.: Comparison of human and machine scoring of essays: differences by gender, ethnicity, and country. Appl. Meas. Educ. 25(1), 27–40 (2012)
Buolamwini, J., Gebru, T.: Gender shades: intersectional accuracy disparities in commercial gender classification. In: Proceedings of the 1st Conference on Fairness, Accountability and Transparency, pp. 77–91 (2018)
Castelnovo, A., Crupi, R., Greco, G., Regoli, D., Penco, I.G., et. al.: A clarification of the nuances in the fairness metrics landscape. Sci. Rep. 12(1), 4209 (2022)
Chen, J., Fang, B., Zhang, H., Xue, X.: A systematic review for MOOC dropout prediction from the perspective of machine learning. Interact. Learn. Environ. 1–14 (2022)
Gardner, J., Brooks, C., Baker, R.: Evaluating the fairness of predictive student models through slicing analysis. In: Proceedings of the 9th International Conference on Learning Analytics & Knowledge, pp. 225–234 (2019)
Idowu, J.A.: Debiasing education algorithms. Int. J. Artif. Intell. Educ. 1–31 (2024)
Kai, S., et al.: Predicting student retention from behavior in an online orientation course. In: Proceedings of the International Conference on Educational Data Mining, pp. 250–255 (2017)
Kizilcec, R.F., Lee, H.: Algorithmic fairness in education. In: The Ethics of Artificial Intelligence in Education, pp. 174–202. Routledge (2022)
Lembani, R., Gunter, A., Breines, M., Dalu, M.T.B.: The same course, different access: the digital divide between urban and rural distance education students in South Africa. J. Geogr. High. Educ. 44(1), 70–84 (2020)
Li, X., Song, D., Han, M., Zhang, Y., Kizilcec, R.F.: On the limits of algorithmic prediction across the globe. arXiv preprint arXiv:2103.15212 (2021)
Littenberg-Tobias, J., Reich, J.: Evaluating access, quality, and equity in online learning: a case study of a MOOC-based blended professional degree program. Internet High. Educ. 47, 100759 (2020)
Moreno-Marcos, P.M., Alario-Hoyos, C., Muñoz-Merino, P.J., Kloos, C.D.: Prediction in MOOCs: a review and future research directions. IEEE Trans. Learn. Technol. 12(3), 384–401 (2018)
Naismith, B., Han, N.R., Juffs, A., Hill, B., Zheng, D.: Accurate measurement of lexical sophistication in ESL with reference to learner data. In: Proceedings of the 11th International Conference on Educational Data Mining, pp. 259–265 (2018)
Rzepka, N., Simbeck, K., Müller, H., Pinkwart, N.: Fairness of in-session dropout prediction. In: Proceedings of the International Conference on Computer Supported Education, pp. 316–326 (2022)
Sha, L., Raković, M., Das, A., Gašević, D., Chen, G.: Leveraging class balancing techniques to alleviate algorithmic bias for predictive tasks in education. IEEE Trans. Learn. Technol. 15(4), 481–492 (2022)
Sylla, K., Nkwetchoua, G., Bouchet, F.: How does the use of open digital spaces impact students success and dropout in a virtual university? In: Proceedings of the Conference on Cognition and Exploratory Learning in Digital Age, pp. 251–258 (2022)
Verger, M., Lallé, S., Bouchet, F., Luengo, V.: Is your model “MADD”? A novel metric to evaluate algorithmic fairness for predictive student models. In: Proceedings of the International Conference on Educational Data Mining, pp. 91–102 (2023)
Wang, Z., Zechner, K., Sun, Y.: Monitoring the performance of human and automated scores for spoken responses. Lang. Test. 35(1), 101–120 (2018)
Xing, W., Du, D.: Dropout prediction in MOOCs: using deep learning for personalized intervention. J. Educ. Comput. Res. 57(3), 547–570 (2019)
Yu, R., Lee, H., Kizilcec, R.F.: Should college dropout prediction models include protected attributes? In: Proceedings of the ACM Conference on Learning@Scale, pp. 91–100 (2021)
Yudelson, M., Fancsali, S., Ritter, S., Berman, S., Nixon, T., Joshi, A.: Better data beats big data. In: Proceedings of the International Conference on Educational Data Mining (2014)
Zambrano, A.F., Zhang, J., Baker, R.S.: Investigating algorithmic bias on Bayesian knowledge tracing and carelessness detectors. In: Proceedings of the 14th International Conference on Learning Analytics & Knowledge, pp. 349–359 (2024)
Acknowledgement
This work was supported by the Sorbonne Center for Artificial Intelligence (SCAI) and the Direction du numérique pour l’éducation (DNE) of the French Ministry of Education. The opinions expressed in this paper are those of the authors and do not necessarily reflect those of DNE and SCAI. We thank Rémi Bachelet and the MOOC GdP’s team for the data collection.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lallé, S., Bouchet, F., Verger, M., Luengo, V. (2024). Fairness of MOOC Completion Predictions Across Demographics and Contextual Variables. In: Olney, A.M., Chounta, IA., Liu, Z., Santos, O.C., Bittencourt, I.I. (eds) Artificial Intelligence in Education. AIED 2024. Lecture Notes in Computer Science(), vol 14829. Springer, Cham. https://doi.org/10.1007/978-3-031-64302-6_27
Download citation
DOI: https://doi.org/10.1007/978-3-031-64302-6_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-64301-9
Online ISBN: 978-3-031-64302-6
eBook Packages: Computer ScienceComputer Science (R0)