A Formal Proof of a Paradox Associated with Cohen’s Kappa
- 249 Downloads
Suppose two judges each classify a group of objects into one of several nominal categories. It has been observed in the literature that, for fixed observed agreement between the judges, Cohen’s kappa penalizes judges with similar marginals compared to judges who produce different marginals. This paper presents a formal proof of this phenomenon.
KeywordsInter-rater reliability Nominal agreement Rearrangement inequality Marginal homogeneity Marginal asymmetry
Unable to display preview. Download preview PDF.
- DE MAST, J. (2007), “Agreement and Kappa-Type Indices,” The American Statistician, 61, 149–153.Google Scholar
- DOU, W., REN, Y., WU, Q., RUAN, S., CHEN, Y., BLOYET, D., and CONSTANS, J.-M. (2007), “Fuzzy Kappa for the Agreement Measure of Fuzzy Classifications,” Neurocomputing, 70, 726–734.Google Scholar
- SIM, J., and WRIGHT, C.C. (2005). The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements, Physical Therapy, 85, 257–268.Google Scholar