Abstract
We wish to study inter-rater agreement comparing groups of observers who express their ratings on a discrete or ordinal scale. The starting point is that of defining what we mean by “agreement”. Given d observers, let the scores they assign to a given statistical unit be expressed as a d-vector in the real space. We define a deterministic ordering among these vectors, which expresses the degree of the raters’ agreement. The overall scoring of the raters on the sample space will be a d-dimensional random vector. We then define an associated partial ordering among the random vectors of the ratings, illustrate a number of its properties, and look at order-preserving functions (agreement measures). In this paper we also show how to test the hypothesis of greater agreement against the unrestricted hypothesis, and the hypothesis of equal agreement against the hypothesis that an agreement ordering holds. The test is applied to real data on two medical observers rating clinical guidelines.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Banerjee M, Capozzoli M, McSweeney L, Sinha D (1999) Beyond kappa: a review of interrater agreement measures. Can J Stat 27(1):3–23
Barlow RE, Bartholomew DJ, Bremner DJ, Brunk HD (1972) Statistical inference under order restrictions. Wiley, New York
Bartholomew DJ (1959) A test of homogeneity for ordered alternatives. Biometrika 46:36–48
Bartolucci F, Forcina A (2002) Extended rc association models allowing for order restrictions and marginal modelling. J Am Stat Assoc 97:1192–1199
Bishop YMM, Fienberg SE, Holland PW (1975) Discrete multivariate analyses: theory and practice. MIT Press, Cambridge
Cicchetti DV (1972) A new measure of agreement between rank ordered variables. Proc Am Psychol Assoc 7:17–18
Cohen J (1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20:37–46
Cohen J (1968) Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull 70:213–220
Colombi R, Forcina A (2000) Modellizzazione di dati discreti con vincoli di uguaglianza e disuguaglianza. Statistica 2:195–214
Dardanoni V, Forcina A (1998) A unified approach to likelihood inference on stochastic orderings in a nonparametric context. J Am Stat Assoc 93:1112–1123
Giovagnoli A (2002) Stochastic orderings and their use in statistics: the case of association between two variables. In: Proceedings of the XLI scientific meeting of the Italian statistical society, Milan, 5–7 June 2002, pp 95–104
Giovagnoli A, Wynn HP (1985) G-majorization with applications to matrix orderings. Linear Algebra Appl 67:111–135
Giovagnoli A, Marzialetti J, Wynn HP (2007) Bivariate stochastic orderings for unordered categorical variables. In: Pronzato L, Zhigljavsky A (eds) W-Optimality in Design and Statistics. Springer, Heidelberg, pp 81–96
Gourieroux C, Holly A, Monfort A (1982) Likelihood ratio test, Wald test, and Kuhn–Tucker test in linear models with inequality constraints on the regression parameters. Econometrica 50:63–80
Kudo NM (1963) A multivariate analogue of the one-sided test. Biometrika 50:403–418
Marshall AW, Olkin I (1979) Inequalities: theory of majorization and its applications. Academic, New York
Marzialetti J (2006) Lo Studio dell’Agreement mediante gli Ordinamenti. Ph.D. thesis, Department of Statistical Sciences, University of Bologna
Robertson T, Wright FT, Dykstra RL (1988) Order restricted statistical inference. Wiley, New York
Schouten HJA (1982) Measuring pairwise interobserver agreement when all subjects are judged by the same observers. Stat Neerl 36:45–61
Shaked M, Shanthikumar JG (2007) Stochastic orders. Springer, Berlin
Shapiro A (1985) Asymptotic distribution of test statistics in the analysis of moment structures under inequality constraints. Biometrika 72:133–144
Shapiro A (1988) Towards a unified theory of inequality constrained testing in multivariate analysis. Int Stat Rev 56:49–62
Silvapulle MJ, Sen PK (2005) Constrained statistical inference: inequality, order and shape restrictions. Wiley-Interscience, Hoboken
Wolak FA (1991) The local and global nature of hypothesis tests involving inequality constraints in nonlinear models. Econometrics 59:981–995
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Giovagnoli, A., Marzialetti, J. & Wynn, H.P. A new approach to inter-rater agreement through stochastic orderings: the discrete case. Metrika 67, 349–370 (2008). https://doi.org/10.1007/s00184-007-0137-4
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00184-007-0137-4