Sklar’s Omega: A Gaussian copula-based framework for assessing agreement

Abstract

The statistical measurement of agreement—the most commonly used form of which is inter-coder agreement (also called inter-rater reliability), i.e., consistency of scoring among two or more coders for the same units of analysis—is important in a number of fields, e.g., content analysis, education, computational linguistics, sports. We propose Sklar’s Omega, a Gaussian copula-based framework for measuring not only inter-coder agreement but also intra-coder agreement, inter-method agreement, and agreement relative to a gold standard. We demonstrate the efficacy and advantages of our approach by applying both Sklar’s Omega and Krippendorff’s Alpha (a well-established nonparametric agreement coefficient) to simulated data, to nominal data previously analyzed by Krippendorff, and to continuous data from an imaging study of hip cartilage in femoroacetabular impingement. Application of our proposed methodology is supported by our open-source R package, sklarsomega, which is available for download from the Comprehensive R Archive Network. The package permits users to apply the Omega methodology to nominal scores, ordinal scores, percentages, counts, amounts (i.e., non-negative real numbers), and balances (i.e., any real number); and can accommodate any number of units, any number of coders, and missingness. Classical inference is available for all levels of measurement while Bayesian inference is available for continuous outcomes only.

Appendices

Appendix A

Here we briefly introduce our R package, sklarsomega, version 3.0 of which is available for download from the Comprehensive R Archive Network.

R package sklarsomega

We introduce our R package by way of a brief usage example. Additional examples are provided in the package documentation.

We apply our Bayesian methodology to a subset of the cartilage data, assuming first a $\textsc {Laplace}(\mu ,\sigma )$ and then a $\textsc {T}(\nu ,\mu )$ marginal distribution. First we load the cartilage data, which are included in the package.

We see that sampling terminated when 4,000 samples had been drawn, since that sample size yielded $\widehat{\text {cv}}_j<0.01$ for $j\in \{1,2,3\}$. As a second check we examine the plot given in Fig. 4, which shows the estimated posterior mean for $\omega$ as a function of sample size. The estimate evidently stabilized after approximately 2,500 samples had been drawn.

Fig. 4

A plot of estimated posterior mean versus sample size for $\omega$, having assumed a Laplace marginal distribution

The proposal standard deviations (1 for $\mu$, 0.1 for $\sigma$, and 0.2 for $\omega$) led to sensible acceptance rates of 40%, 60%, and 67%.

For a t marginal distribution only 3,000 samples were required.

Note that the Laplace model yielded a much smaller value of DIC, and hence a very small relative likelihood for the t model.

Much additional functionality is supported by package sklarsomega, e.g., plotting, simulation, influence statistics. And we note that computational efficiency is supported by our use of sparse-matrix routines (Furrer and Sain 2010) and a clever bit of Fortran code (Genz 1992) for the CML method. Future versions of the package will employ C++ (Eddelbuettel and Francois 2011).