Abstract
Traditional efforts around improving assessment often center on the teacher as the evaluator of work rather than the students. These assessment efforts typically focus on measuring learning rather than stimulating, promoting, or producing learning in students. This paper summarizes a study of a large sample of undergraduate students (n = 550) in an entry-level design-thinking course who engaged with Adaptive Comparative Judgment (ACJ), a form of assessment, as a learning mechanism. Following random assignment into control and treatment sections, students engaged in identical activities with the exception of a 20-minute intervention we call learning by evaluating (LbE). Prior to engaging in a Point Of View (POV) creation activity, treatment group students engaged in LbE by viewing pairs of previously-collected POV statements through ACJ; in each case they viewed two POV statements side-by-side and selected the POV statement they believed was better. Following this experience, students created their own POV statements and then the final POV statements, from both the control and treatment students, were collected and evaluated by instructors using ACJ. In addition, qualitative data consisting of student comments, collected during ACJ comparisons, were coded by the researchers to further explore the potential for the students to use class knowledge while engaging in the LbE review of peer work. Both the quantitative and qualitative data sets were analyzed to investigate the impact of the LbE activity. Consistent with other ACJ research findings, significant positive learning gains were found for students who engaged in the intervention. Researchers also noted that these findings did not indicate the actual quality of the assignments, meaning the while students who engaged in the LbE intervention were better than their peers, they were not necessarily “good” at the assignment themselves. Discussion of these findings and areas for further inquiry are presented.
Similar content being viewed by others
References
Baniya, S., Chesley, A., Mentzer, N., Bartholomew, S., Moon, C., & Sherman, D. (2019). Using adaptive comparative judgment in writing assessment: An investigation of reliability among interdisciplinary evaluators. Journal of Technology Studies, 45(2), 24–45.
Bartholomew, S. R. (2017). Assessing open-ended design problems. Technology and Engineering Education Teacher, 76(6), 13–17.
Bartholomew, S. R. (2019). Teacher led assessment of learning to student led assessment as learning. Beyond multiple choice conference, Washington DC.
Bartholomew, S. R., & Jones, M. D. (2020). A systematic review of research of adaptive comparative judgement (ACJ) in higher education. International Journal of Technology and Design Education. https://doi.org/10.21061/ctete-rms.v1.c.1
Bartholomew, S. R., Strimel, G. J., Garcia Bravo, E., Zhang, L., & Yoshikawa, E. (2018). Formative feedback for improved student performance through adaptive comparative judgment. Paper presented at the paper presented at the 125th ASEE conference, Salt Lake City, Utah.
Bartholomew, S. R., Strimel, G. S., & Yoshikawa, E. (2018b). Using adaptive comparative judgment for student formative feedback and learning during a middle school open- ended design challenge. International Journal of Technology and Design Education, 29(2), 363–385.
Bartholomew, S. R., & Yoshikawa, E. (2018). A systematic review of research around Adaptive Comparative Judgment (ACJ) in K-16 education. 2018 CTETE Monograph Series. https://doi.org/10.21061/ctete-rms.v1.c.1.
Bartholomew, S. R., Zhang, L., Garcia Bravo, E., & Strimel, G. J. (2019). A tool for formative assessment and learning in a graphics design course: adaptive comparative judgement. The Design Journal, 22(1), 73–95.
Black, P., & Wiliam, D. (2005). Inside the black box: Raising standards through classroom assessment. Granada Learning.
Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability (formerly: Journal of Personnel Evaluation in Education), 21(1), 5–31.
Black, P., & Wiliam, D. (2018). Classroom assessment and pedagogy. Assessment in Education: Principles, Policy & Practice, 25(6), 551–575.
Box, C. (2019). Formative assessment in United States classrooms. Springer Science + Business Media.
Bramley, T. (2015). Investigating the reliability of Adaptive Comparative Judgment. Cambridge Assessment, Cambridge, 36
Brown, S. (2014). Learning, teaching and assessment in higher education: Global perspectives. Macmillan International Higher Education.
Canty, D. (2012). The impact of holistic assessment using adaptive comparative judgment of student learning. Limerick: University of Limerick.
Denson, C. D., Buelin, J. K., Lammi, M. D., & D’Amico, S. (2015). Developing instrumentation for assessing creativity in engineering design. Journal of Technology Education, 27(1), 23–40.
Ellis, A. K. (2017). Teaching, learning, and assessment together: Reflective assessments for elementary classrooms. Routledge.
Gielen, M., & de Wever, B. (2015). Structuring peer assessment: Comparing the impact of the degree of structure on peer feedback content. Computers in Human Behavior, 52, 315–325.
Giloi, S., & du Toit, P. (2013). Current approaches to the assessment of graphic design in a higher education. International Journal of Art & Design Education, 32(2), 256–268.
Hamer, J., Purchase, H., Luxton-Reilly, A., & Denny, P. (2015). A comparison of peer and tutor feedback. Assessment & Evaluation in Higher Education, 40(1), 151–164.
Jones, I., & Alcock, L. (2014). Peer assessment without assessment criteria. Studies in Higher Education, 39(10), 1774–1787.
Jurado, J. A. (2011). Group projects in interior design studio classes: Peer feedback benefits. Journal of Family and Consumer Sciences, 103(1), 34.
Kimbell, R. (2007). E-assessment in project e-scape. Design & Technology Education: An International Journal, 12(2), 66–76.
Kimbell, R. (2012a). Evolving project e-scape for national assessment. International Journal of Technology and Design Education, 22(2), 135–155.
Kimbell, R. (2012b). The origins and underpinning principles of e-scape. International Journal of Technology and Design Education, 22, 123–124.
Kimbell, R. (2018). Constructs of quality and the power of holism. Paper presented at the PATT35, Athlone Institute of Technology, Athlone, Ireland.
Lau, A. M. S. (2016). ‘Formative good, summative bad?’—A review of the dichotomy in assessment literature. Journal of Further and Higher Education, 40(4), 509–525.
Li, L., & Gao, F. (2016). The effect of peer assessment on project performance of students at different learning levels. Assessment & Evaluation in Higher Education, 41(6), 885–900.
Li, L., Liu, X., & Steckelberg, A. L. (2010). Assessor or assessee: How student learning improves by giving and receiving peer feedback. British Journal of Educational Technology, 41(3), 525–536.
López-Pastor, V., & Sicilia-Camacho, A. (2017). Formative and shared assessment in higher education. Lessons learned and challenges for the future. Assessment & Evaluation in Higher Education, 42(1), 77–97.
McCarthy, J. (2017). Enhancing feedback in higher education: Students’ attitudes towards online and in-class formative assessment feedback models. Active Learning in Higher Education, 18(2), 127–141.
McGarr, O., & Clifford, A. M. (2013). ‘Just enough to make you take it seriously’: Exploring students’ attitudes towards peer assessment. Higher Education, 65(6), 677–693.
Mehalik, M. M., Doppelt, Y., & Schuun, C. D. (2008). Middle-school science through design-based learning versus scripted inquiry: Better overall science concept learning and equity gap reduction. Journal of Engineering Education, 97(1), 71–85.
Nicol, D., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: A peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102–122.
Penuel, W. R., Harris, C. J., & DeBarger, A. H. (2015). Implementing the next generation science standards. Phi Delta Kappan, 96(6), 45–49.
Pollitt, A. (2004). Let’s stop marking exams. Retrieved from http://www.cambridgeassessment.org.uk/images/109719-let-s-stop-marking-exams.pdf.
Pollitt, A. (2012). The method of adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 19(3), 281–300.
Pollitt, A. (2015). On ‘Reliability’ bias in ACJ. Cambridge Exam Research. Retrieved 23, April 2020 from https://www.researchgate.net/publication/283318012_On_’Reliability’_bias_in_ACJ.
Potter, T., Englund, L., Charbonneau, J., MacLean, M. T., Newell, J., & Roll, I. (2017). ComPAIR: A new online tool using adaptive comparative judgement to support learning with peer feedback. Teaching & Learning Inquiry, 5(2), 89–113.
Rangel-Smith, C., & Lynch, D. (2018). Addressing the issue of bias in the measurement of reliability in the method of Adaptive Comparative Judgment. In 36th pupils’ attitudes towards technology conference, Athlone, Ireland, pp. 378–387.
Saldaña, J. (2015). The coding manual for qualitative Researchers. Thousand Oaks, CA: SAGE.
Seery, N., & Canty, D (2017). Assessment and learning: The proximal and distal effects of comparative judgment. Handbook of Technology Education, Springer. DOI: 10.1007/978-3-319-38889-2_54-1.
Sluijsmans, D., Dochy, F., & Moerkerke, G. (1998). Creating a learning environment by using self-, peer- and co-assessment. Learning Environments Research, 1(3), 293–319.
Stefani, L. A. J. (1994). Peer, self and tutor assessment: Relative reliabilities. Studies in Higher Education, 19, 69–75.
Strimel, G. J., Bartholomew, S. R., Purzer, S., Zhang, L., & Ruesch, E. Y. (2020). Informing engineering design through adaptive comparative judgment. European Journal of Engineering Education. https://doi.org/10.1080/03043797.2020.1718614.
Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–286.
Verhavert, S., Bouwer, R., Donche, V., & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(5), 541–562.
Walker, M. (2015). The quality of written peer feedback on undergraduates’ draft answers to an assignment, and the use made of the feedback. Assessment & Evaluation in Higher Education, 40(2), 232–247.
Wells, J. G. (2016). PIRPOSAL Model of Integrative STEM Education: Conceptual and Pedagogical Framework for Classroom Implementation. Technology and Engineering Teacher, 75(6), 12–19.
Zhang, L. (2019). Investigating differences in formative critiquing between instructors and students in graphic design. Doctoral dissertation, Purdue University Libraries.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Bartholomew, S.R., Mentzer, N., Jones, M. et al. Learning by evaluating (LbE) through adaptive comparative judgment. Int J Technol Des Educ 32, 1191–1205 (2022). https://doi.org/10.1007/s10798-020-09639-1
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10798-020-09639-1