Are ordinal rating scales better than percent ratings? a statistical and “psychological” view

Abstract

Disease incidence and severity are often assessed by either an ordinal rating scale, e.g., with scores from 1 to 9, or a percentage rating scale. This paper compares three different rating scales regarding accuracy, precision, and time needed for scoring. Pictograms of mildew diseased cereal leaves were generated following a right skewed beta-distribution. Persons with different rating experience were asked to rate the leaves on three different scales: two different percentage scales [1%-steps (P1) and 5%-steps (P5)] and an ordinal 9-point rating scale (R9) where thresholds followed a logarithmic pattern with respect to the underlying percentage scale. A transformed value of the estimated disease severity as well as the transformed time needed to estimate per leaf was documented and evaluated using mixed models. In most cases both percent ratings performed better than the ordinal rating scale. For the time needed per leaf by the untrained group, method R9 was better. With the trained group P5 performed better than both other methods. The raters mostly preferred R9, especially when untrained. Nevertheless, the results suggest that P5 can be recommended in terms of accuracy.

Appendix

Covariance structures for [ $(\alpha \beta )_{ij}$, $\delta _{ij}$] were

FA0(1): $\text{var} \left( {\begin{array}{*{20}c} {(\alpha \beta )_{ij} } \\ {\delta _{ij} } \\ \end{array} } \right) = \left[ {\begin{array}{*{20}c} {\lambda _1^2 } & {\lambda _1 \lambda _2 } \\ {\lambda _1 \lambda _2 } & {\lambda _2^2 } \\ \end{array} } \right]$,

UN: $\text{var} \left( {\begin{array}{*{20}c} {(\alpha \beta )_{ij} } \\ {\delta _{ij} } \\ \end{array} } \right) = \left[ {\begin{array}{*{20}c} {\sigma _1^2 } & {\sigma _{12} } \\ {\sigma _{12} } & {\sigma _2^2 } \\ \end{array} } \right]$ (SAS Institute Inc 1999).