Clinical Agreement in Qualitative Measurements

Abstract

Agreement between raters on a categorical scale is not only a subject of scientific research but also a problem frequently encountered in practice. For example, in psychiatry, the mental illness of a subject may be judged as light, moderate or severe. Inter- and intra-rater agreement is a prerequisite for the scale to be implemented in routine use. Agreement studies are therefore crucial in health, medicine and life sciences. They provide information about the amount of error inherent to any diagnostic, score or measurement (e.g. disease diagnostic or implementation quality of health promotion interventions). The kappa-like coefficients (intraclass kappa, Cohen’s kappa and weighted kappa), usually used to assess agreement between or within raters on a categorical scale, are reviewed in this chapter with emphasis on the interpretation and the properties of these coefficients.

Appendix: Variance of the Kappa Coefficients

The sample variance of Cohen’s kappa coefficient using the delta method is given by

$$ \operatorname{var}(\hat{\kappa})=\frac{{{p_\mathrm{{o}}}(1-{p_\mathrm{{o}}})}}{{N{{{(1-{p_\mathrm{{e}}})}}^2}}}+\frac{{2({p_\mathrm{{o}}}-1)({C_1}-2{p_\mathrm{{o}}}{p_\mathrm{{e}}})}}{{N{{{(1-{p_\mathrm{{e}}})}}^3}}}+\frac{{{{{({p_\mathrm{{o}}}-1)}}^2}({C_2}-4p_\mathrm{{e}}^2)}}{{N{{{(1-{p_\mathrm{{e}}})}}^4}}} $$

(1.5)

where

$$ {C_1}=\sum\limits_{j=1}^K {p_{jj }}({p_{j. }}+{p_{.j }})\quad \mathrm{and}\quad {{\it C}_2}=\sum\limits_{{\it j}=1}^{\it K} \sum\limits_{{\it k}=1}^{\it K} {\it p_{jk }}{{({\it p_{.j }}+{\it p_{k. }})}^2}. $$

With the additional assumption of no rater bias, the sample variance simplifies to

$$ \operatorname{var}({{\hat{\kappa}}_{\mathrm{I}}})=\frac{1}{{N{{{(1-{C_3})}}^2}}}\left\{ \begin{gathered} \sum\limits_{j=1}^K {p_{jj }}[1-4{{\bar{p}}_j}(1-{{\hat{\kappa}}_{\mathrm{I}}})] \hfill \\ +{{(1-{{\hat{\kappa}}_{\mathrm{I}}})}^2}\sum\limits_{j=1}^K \sum\limits_{k=1}^K {p_{jk }}{{({{\bar{p}}_j}+{{\bar{p}}_k})}^2}-{{[{{\hat{\kappa}}_{\mathrm{I}}}-{C_3}(1-{{\hat{\kappa}}_{\mathrm{I}}})]}^2} \hfill \\ \end{gathered} \right\} $$

(1.6)

where ${{\bar{p}}_j}=({p_{j. }}+{p_{.j }})/2$ and ${C_3}=\sum\nolimits_{j=1}^K {{\bar{p}}_j}$ ($j,k=1,\cdots, K$).

The two sided $(1-\alpha )$ confidence interval for $\kappa$ is then determined by $\hat{\kappa}\pm {Q_z}(1-\alpha /2)\sqrt{{\operatorname{var}(\hat{\kappa})}}$, where ${Q_z}(1-\alpha /2)$ is the $\alpha /2$ upper percentile of the standard Normal distribution.

The sample variance of the weighted kappa coefficient obtained by the delta method is

$$ \operatorname{var}({{\hat{\kappa}}_{\mathrm{w}}})=\frac{1}{{N{{{\left( {1-{p_{\mathrm{ew}}}} \right)}}^4}}}\left\{ \begin{gathered} \sum\limits_{j=1}^K \sum\limits_{k=1}^K {p_{jk }}{{\left[ {{w_{jk }}\left( {1-{p_{\mathrm{ew}}}} \right)-\left( {{{\overline{w}}_{j. }}+{{\overline{w}}_{.k }}} \right)\left( {1-{p_{\mathrm{ow}}}} \right)} \right]}^2} \hfill \\ -{{\left( {{p_{\mathrm{ow}}}{p_{\mathrm{ew}}}-2{p_{\mathrm{ew}}}+{p_{\mathrm{ow}}}} \right)}^2} \hfill \\ \end{gathered} \right\} $$

(1.7)

where ${{\overline{w}}_{.j }}=\sum\nolimits_{m=1}^K {w_{mj }}{p_{m. }}$ and ${{\overline{w}}_{k. }}=\sum\nolimits_{s=1}^K {w_{ks }}{p_{.s }}$.