The Australian Educational Researcher

, Volume 43, Issue 3, pp 273–288 | Cite as

A study of the use of pairwise comparison in the context of social online moderation

  • Pina Tarricone
  • C. Paul Newhouse


Traditional moderation of student assessments is often carried out with groups of teachers working face-to-face in a specified location making judgements concerning the quality of representations of achievement. This traditional model has relied little on modern information communications technologies and has been logistically challenging. We argue that social online moderation, coupled with the use of analytical and pairwise scoring methods and technologies, can provide better moderation outcomes and highly valuable professional learning experiences improving teachers’ understandings of assessment standards. This paper reports on a component of a study involving Visual Arts teachers from rural schools making comparative judgements of digitised student artworks. We report the teachers’ observations of the social online moderation processes, including the quality and standard of the digitised artworks, the effectiveness of the pairwise comparison process, the functionality of the online tools, and the concept of using online scoring for moderation and standard setting purposes.


Comparative judgement Paired comparison Pairwise comparison Social online moderation Moderation 



The study discussed in this paper was the work of a research team led by Paul Newhouse and included researchers Jeremy Pagram, Lisa Paris, Mark Hackling, Martin Cooper, Pina Tarricone, Alistair Campbell, Alun Price and many research assistants. The work of everyone in this team, particularly Martin Cooper and Pina Tarricone, and the teachers and students involved, contributed to the research outcomes presented in this paper.


  1. Adie, L. E. (2011). An investigation into online moderation. Assessment Matters, 3, 5–27.Google Scholar
  2. Adie, L. E. (2013a). The development of shared understandings of assessment policy: Travelling between global and local contexts. Journal of Education Policy, 29(4), 1–14.Google Scholar
  3. Adie, L. E. (2013b). The development of teacher assessment identity through participation in online moderation. Assessment in Education: Principles, Policy & Practice, 20(1), 91–106.CrossRefGoogle Scholar
  4. Adie, L. E., Klenowski, V., & Wyatt-Smith, C. (2012). Towards an understanding of teacher judgement in the context of social moderation. Educational Review, 64(2), 223–240.CrossRefGoogle Scholar
  5. Andrich, D. (1988). Rasch models for measurement. Newbury Park: Sage Publications.Google Scholar
  6. Heldsinger, S., & Humphry, S. M. (2010). Using the method of pairwise comparison to obtain reliable teacher assessments. The Australian Educational Researcher, 37(2), 1–19.CrossRefGoogle Scholar
  7. Hipkins, R., & Robertson, S. (2012). The complexities of moderating student writing in a community of practice. Assessment Matters, 4, 30–52.Google Scholar
  8. Humphry, S. M., & McGrane, J. A. (2015). Equating a large-scale writing assessment using pairwise comparisons of performances. The Australian Educational Researcher, 42, 443–460.CrossRefGoogle Scholar
  9. Humphry, S. M., Wray, W. H., & Wray, F. W. (2013–2015). Pair-Wise Web Software. Perth, Western Australia: The University of Western Australia.Google Scholar
  10. Jones, I., & Alcock, L. (2012). Summative peer assessment of undergraduate calculus using adaptive comparative judgement. In P. Iannone & A. Simpson (Eds.), Mapping university mathematics assessment practices (pp. 63–74). East Anglia: University of East Anglia.Google Scholar
  11. Klenowski, V. (2013). Towards improving public understanding of judgement practice in standards-referenced assessment: An Australian perspective. Oxford Review of Education, 39(1), 36–51.CrossRefGoogle Scholar
  12. Klenowski, V., & Wyatt-Smith, C. (2010). Standards-driven reform years 1-10: Moderation an optional extra? The Australian Educational Researcher, 37(2), 21–39.CrossRefGoogle Scholar
  13. Malone, L., Long, K., & De Lucchi, L. (2004). All things in moderation. Science and Children, 41(5), 30–34.Google Scholar
  14. Newhouse, C. P. (2014). Using digital representations of practical production work for summative assessment. Assessment in Education: Principles, Policy & Practice, 21, 205–220.CrossRefGoogle Scholar
  15. Newhouse, C. P., & Tarricone, P. (2014). Digitizing practical production work for high-stakes assessments. Canadian Journal of Learning and Technology, 40(2), 1–17.Google Scholar
  16. Pollitt, A. (2012). The method of adaptive comparative judgement. Assessment in Education: principles, policy & practice, 19(3), 281–300.CrossRefGoogle Scholar
  17. Poskitt, J., & Mitchell, K. (2012). New Zealand teachers’ overall teacher judgements (OTJs): Equivocal or unequivocal? Assessment Matters, 4, 53–75.Google Scholar
  18. Rasch, G. (1961). On general laws and the meaning of measurement in psychology. Paper presented at the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Statistical Laboratory Proceedings, June 20–July 30 1960, University of California.Google Scholar
  19. Smith, C. (2012). Why should we bother with assessment moderation? Nurse Education Today, 32, 45–48.CrossRefGoogle Scholar
  20. Tarricone, P., & Cooper, M. G. (2014). Using Rasch measurement to improve analytical marking keys. Assessment Matters, 6, 86–111.Google Scholar
  21. Tarricone, P., & Newhouse, C. P. (in publication). Using comparative judgement and online technologies in the assessment and measurement of creative performance and capability. RUSC Universities and Knowledge Society Journal: A bilingual refereed research e-journal in e-learning, university and network society.Google Scholar
  22. Thurstone, L. L. (1927). A law of comparative judgement. Psychological Review, 34, 278–286.Google Scholar
  23. Wilson, M. (2004). Assessment, accountability and the classroom: A community of judgement. In M. Wilson (Ed.), Towards coherence between classroom assessment and accountability (pp. 1–19). Chicago, IL: University of Chicago Press.Google Scholar
  24. Wilson, M. (2005). Constructing measures: An item response modelling approach. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  25. Wyatt-Smith, C., Klenowski, V., & Gunn, S. (2010). The centrality of teachers’ judgement practice in assessment: A study of standards in moderation. Assessment in Education: Principles, Policy & Practice, 17(1), 59–75.CrossRefGoogle Scholar

Copyright information

© The Australian Association for Research in Education, Inc. 2016

Authors and Affiliations

  1. 1.Centre for Schooling and Learning Technologies (CSaLT), School of EducationEdith Cowan UniversityMount LawleyAustralia

Personalised recommendations