Detecting and Measuring Rater Effects in Interpreting Assessment: A Methodological Comparison of Classical Test Theory, Generalizability Theory, and Many-Facet Rasch Measurement

Abstract

Human raters play an important role in interpreting assessment. Their evaluative judgment of the quality of interpreted renditions produces quantitative measures (e.g., scores, ratings, marks, ranks) that form the basis of relevant decision-making (e.g., program admission, professional certification). Previous research in the field of language testing finds that human raters may be inconsistent over time, excessively harsh or lenient, and biased against a particular group of test candidates, a certain type of tasks and a given assessment criterion. Such undesirable phenomena (i.e., rater inconsistency, rater severity/leniency, and rater bias) are collectively known as rater effects. Their presence could lead to unreliable, invalid, and unfair assessments. It is therefore of importance to investigate possible rater effects in interpreting assessment. Although a number of statistical indices can be computed to measure rater effects, there has been no systematic attempt to compare their applicability and utility. Against this background, the current study aims to compare three psychometric approaches, namely, classical test theory, generalizability theory, and many-facet Rasch measurement, to detecting and measuring rater effects. Our analysis is based on the data from a previous assessment of English-to-Chinese simultaneous interpreting in which a total of nine raters were involved. Through our comparison, we hope that interpreting researchers and testers could obtain an in-depth understanding of statistical information generated by these approaches, and be able to make informed decisions in selecting an analytic approach commensurate with their local assessment needs.

Appendix 1 The Analytic Rating Scale

Band/scoring criteria	Information completeness (InfoCom)	Fluency of delivery (FluDel)	Target language quality (TLQual)
Band 4 (Score range: 7–8)	A substantial amount of original messages delivered (i.e., > 90%), with a few number of deviations, inaccuracies, and minor/major omissions	Delivery on the whole fluent, containing a few disfluencies such as (un)filled pauses, long silence, fillers and/or excessive repairs	Target language idiomatic and on the whole correct, with only a few instances of unnatural expressions and grammatical errors
Band 3 (Score range: 5–6)	Majority of original messages delivered (i.e., 60–70%), with a small number of deviations, inaccuracies, and minor/major omissions	Delivery on the whole generally fluent, containing a small number of disfluencies	Target language generally idiomatic and on the whole mostly correct, with a small amount of instances of unnatural expressions and grammatical errors
Band 2 (Score range: 3–4)	About half of original messages delivered (i.e., 40–50%), with many instances of deviations, inaccuracies, and minor/major omissions	Delivery rather fluent. Acceptable, but with regular disfluencies	Target language to a certain degree both idiomatic and correct. Acceptable, but contains many instances of unnatural expressions and grammatical errors
Band 1 (Score range: 1–2)	A small portion of original messages delivered (i.e., < 30%), with frequent occurrences of deviations, inaccuracies, and minor/major omissions, to such a degree that listeners may doubt the integrity of renditions	Delivery lacks fluency. It is frequently hampered by disfluencies, to such a degree that they may impede comprehension	Target language stilted, lacking in idiomaticity, and containing frequent grammatical errors, to such a degree that it may impede comprehension