Psychometrika

, Volume 60, Issue 2, pp 199–220

General estimators for the reliability of qualitative data

Article

DOI: 10.1007/BF02301413

Cite this article as:
Cooil, B. & Rust, R.T. Psychometrika (1995) 60: 199. doi:10.1007/BF02301413

Abstract

We study a proportional reduction in loss (PRL) measure for the reliability of categorical data and consider the general case in which each ofN judges assigns a subject to one ofK categories. This measure has been shown to be equivalent to a measure proposed by Perreault and Leigh for a special case when there are two equally competent judges, and the correct category has a uniform prior distribution. We consider a general framework where the correct category is assumed to have an arbitrary prior distribution, and where classification probabilities vary by correct category, judge, and category of classification. In this setting, we consider PRL reliability measures based on two estimators of the correct category—the empirical Bayes estimator and an estimator based on the judges' consensus choice. We also discuss four important special cases of the general model and study several types of lower bounds for PRL reliability.

Key words

empirical Bayesintercoder reliabilitylatent classproportional reduction in loss

Copyright information

© The Psychometric Society 1995

Authors and Affiliations

  1. 1.Owen Graduate School of ManagementVanderbilt UniversityNashville