Quality & Quantity

, Volume 47, Issue 5, pp 2959–2982

Factors affecting intercoder reliability: a Monte Carlo experiment


DOI: 10.1007/s11135-012-9745-9

Cite this article as:
Feng, G.C. Qual Quant (2013) 47: 2959. doi:10.1007/s11135-012-9745-9


Although it has long been a consensus that intercoder reliability is crucial to the validity of a content analysis study, the choice among them has been debated. This study reviewed and empirically tested most popular intercoder reliability indices, aiming to find the most robust index against prevalence and rater bias, by empirically testing their relationships with response surface methodology through a Monte Carlo experiment. It was found that Maxwell’s R.E is superior to Krippendorff’s α, Scott’s π, Cohen’s κ, Ir of Perreault and Leigh, and Gwet’s AC1. More nuanced relationships among prevalence, sensitivity, specificity and the intercoder reliability indices were discovered through response surface plots. Both theoretical and practical implications were also discussed in the end.


Intercoder reliability Prevalence Simulation 

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  1. 1.School of CommunicationHong Kong Baptist UniversityKowloonHong Kong

Personalised recommendations