Abstract
Due to the high number of students enrolled and the relatively small number of available tutors, the assessment of complex assignments is deemed as one of the most critical tasks in Massive Open On-line Courses (MOOCs). Peer assessment is becoming an increasingly popular tool to face this problem and many approaches have been proposed so far to make its outcomes more reliable. A promising approach is FOPA (Fuzzy Ordinal Peer Assessment) that adopts and integrates models coming from Fuzzy Set Theory and Group Decision Making. In this paper we propose a FOPA extension supporting multi-criteria assessment based on rubrics. Students are asked to rank a small number of peer submissions against specified criteria, provided rankings are then transformed in fuzzy preference relations, expanded to obtain missing values and aggregated to establish a global ranking between students’ works with respect to each criterion and globally. The absolute grades of all submissions are then calculated.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Glance, D.G., Forsey, M., Riley, M.: The pedagogical foundations of massive open online courses. First Monday 18(5) (2013)
Bouzidi, L., Jaillet, A.: Can online peer assessment be trusted? Educ. Technol. Soc. 12(4), 257–268 (2009)
Carlson, P.A., Berry, F.C.: Calibrated peer review™ and assessing learning outcomes. In: Proceedings of the 33rd International Conference Frontiers in Education (2003)
Walsh, T.: The peer rank method for peer assessment. In: Proceedings of the 21st European Conference on Artificial Intelligence (2014)
Capuano, N., Caballé, S.: Towards adaptive peer assessment for MOOCs. In: Proceedings of the 10th International Conference on P2P, Parallel, GRID, Cloud and Internet Computing (3PGCIC 2015), Krakow, Poland (2015)
Raman, K., Joachims, T.: Methods for ordinal peer grading. In: Proceedings of the 20th SIGKDD International Conference on Knowledge Discovery and Data Mining (2014)
Capuano, N., Loia, V., Orciuoli, F.: A fuzzy group decision making model for ordinal peer assessment. IEEE Trans. Learn. Technol. 10(2), 247–259 (2017)
Capuano, N., Orciuoli, F.: Application of fuzzy ordinal peer assessment in formative evaluation. In: Proceedings of the 12th International Conference on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC 2017), Barcelona, Spain (2017)
Albano, G., Capuano, N., Pierri, A.: Adaptive peer grading and formative assessment. J. e-Learn. Knowl. Soc. 13(1), 147–161 (2017)
Reddy, Y., Andrade, H.: A review of rubric use in higher education. Assess. Eval. High. Educ. 35(4), 435–448 (2009)
Lan, C.H., Graf, S., Lai, K.R., Kinshuk, K.: Enrichment of peer assessment with agent negotiation. IEEE Trans. Learn. Technol. 4(1), 35–46 (2011)
Capuano, N., Caballé, S., Miguel, J.: Improving peer grading reliability with graph mining techniques. Int. J. Emerg. Technol. Learn. 11(7), 24–33 (2016)
Caragiannis, I., Krimpas, A., Voudouris, A.A.: Aggregating partial rankings with applications to peer grading in massive online open courses. In: Proceedings of the International Conference on Autonomous Agents and Multiagent Systems, Istanbul (2015)
Borda, J.C.: Memoire sur les elections au scrutin, Histoire de l’Académie Royale des Sciences (1781)
Pedrycz, W., Ekel, P., Parreiras, R.: Fuzzy Multicriteria Decision-Making: Models, Methods and Applications. Wiley, Hoboken (2010)
Lu, J., Zhang, G., Ruan, D., Wu, F.: Multi-objective group decision making, methods, software and applications with fuzzy set techniques, World Scientific (2007)
Capuano, N., Chiclana, F., Fujita, H., Herrera-Viedma, E., Loia, V.: Fuzzy group decision making with incomplete information guided by social influence. In: IEEE Transactions of Fuzzy Systems. In Press
Zadeh, L.A.: A computational approach to fuzzy quantifiers in natural languages. Comput. Math Appl. 9, 149–184 (1983)
Wang, Y.M., Fan, Z.P.: Fuzzy preference relations: aggregation and weight determination. Comput. Ind. Eng. 53(1), 163–172 (2007)
Yager, R.R.: Families of OWA operators. Fuzzy Sets Syst. 59(2), 125–148 (1993)
Chiclana, F., Herrera, F., Herrera-Viedma, E.: Integrating three representation models in fuzzy multipurpose decision making based on fuzzy preference relations. Fuzzy Sets Syst. 97(1), 33–48 (1998)
Capuano, N., Chiclana, F., Fujita, H., Herrera-Viedma, E., Loia, V.: Fuzzy rankings for preferences modeling in group decision making. Int. J. Intell. Syst. 33(7), 1555–1570 (2018)
Alonso, S., Chiclana, F., Herrera, F., Herrera-Viedma, E., Alcala-Fdez, J., Porcel, C.: A consistency-based procedure to estimate missing pairwise preference values. Int. J. Intell. Syst. 23(1), 155–175 (2008)
Alonso, S., Herrera-Viedma, E., Chiclana, F., Herrera, F.: Individual and social strategies to deal with ignorance situations in multi-person decision making. Int. J. Inf. Technol. Decis. Making 8(2), 313–333 (2009)
Jonsson, A., Svingby, G.: The use of scoring rubrics: reliability, validity and educational consequences. Educ. Res. Rev. 2(2), 130–144 (2007)
Ekel, P., Queiroz, J., Parreiras, R., Palhares, R.: Fuzzy set based models and methods of multicriteria group decision-making. Nonlinear Anal.: Theor. Methods Appl. 71(12), 409–419 (2009)
Chiclana, F., Herrera-Viedma, E., Herrera, F., Alonso, S.: Some induced ordered weighted averaging operators and their use for solving group decision-making problems based on fuzzy preference relations. Eur. J. Oper. Res. 182(1), 383–399 (2007)
Acknowledgement
This work has been supported by the project colMOOC “Integrating Conversational Agents and Learning Analytics in MOOCs”, co-funded by the European Commission within the Erasmus + Knowledge Alliances program (ref. 588438-EPP-1-2017-1-EL-EPPKA2-KA).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Capuano, N., Caballé, S. (2019). Multi-criteria Fuzzy Ordinal Peer Assessment for MOOCs. In: Xhafa, F., Barolli, L., Greguš, M. (eds) Advances in Intelligent Networking and Collaborative Systems. INCoS 2018. Lecture Notes on Data Engineering and Communications Technologies, vol 23. Springer, Cham. https://doi.org/10.1007/978-3-319-98557-2_34
Download citation
DOI: https://doi.org/10.1007/978-3-319-98557-2_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-98556-5
Online ISBN: 978-3-319-98557-2
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)