Rater variables associated with ITER ratings
- 255 Downloads
Advocates of holistic assessment consider the ITER a more authentic way to assess performance. But this assessment format is subjective and, therefore, susceptible to rater bias. Here our objective was to study the association between rater variables and ITER ratings. In this observational study our participants were clerks at the University of Calgary and preceptors who completed online ITERs between February 2008 and July 2009. Our outcome variable was global rating on the ITER (rated 1–5), and we used a generalized estimating equation model to identify variables associated with this rating. Students were rated “above expected level” or “outstanding” on 66.4 % of 1050 online ITERs completed during the study period. Two rater variables attenuated ITER ratings: the log transformed time taken to complete the ITER [β = −0.06, 95 % confidence interval (−0.10, −0.02), p = 0.002], and the number of ITERs that a preceptor completed over the time period of the study [β = −0.008 (−0.02, −0.001), p = 0.02]. In this study we found evidence of leniency bias that resulted in two thirds of students being rated above expected level of performance. This leniency bias appeared to be attenuated by delay in ITER completion, and was also blunted in preceptors who rated more students. As all biases threaten the internal validity of the assessment process, further research is needed to confirm these and other sources of rater bias in ITER ratings, and to explore ways of limiting their impact.
KeywordsPerformance assessment In-training evaluation Illusory superiority Medical students Regression to the mean
- Alicke, M. D., & Govorun, O. (2005). The better-than-average effect. In M. D. Alicke, D. A. Dunning, & J. Krueger (Eds.), The self in social judgment. Studies in self and identity. Hove, NY: Psychology Press.Google Scholar
- Bandiera, G., & Lendrum, D. (2008). Daily encounter cards facilitate competency-based feedback while leniency bias persists. CJEM Canadian Journal of Emergency Medical Care, 10, 44–50.Google Scholar
- LCME (2011) Functions and structure of a medical school: Standards for accreditation of medical education programs leading to the M.D. Degree (May 2011). http://www.lcme.org/functions2011may.pdf (Accessed July 2011).
- Tonesk, X., & Buchanan, R. G. (1987). An AAMC pilot study by 10 medical schools of clinical evaluation of students. Journal of Medical Education, 62, 707–718.Google Scholar