Article

Advances in Health Sciences Education

, Volume 18, Issue 3, pp 375-396

Open Access This content is freely available online to anyone, anywhere at any time.

Workplace-based assessment: raters’ performance theories and constructs

  • M. J. B. GovaertsAffiliated withDepartment of Educational Research and Development, FHML, Maastricht University Email author 
  • , M. W. J. Van de WielAffiliated withDepartment of Work and Social Psychology, FPN, Maastricht University
  • , L. W. T. SchuwirthAffiliated withSchool of Medicine, Flinders University
  • , C. P. M. Van der VleutenAffiliated withDepartment of Educational Research and Development, FHML, Maastricht University
  • , A. M. M. MuijtjensAffiliated withDepartment of Educational Research and Development, FHML, Maastricht University

Abstract

Weaknesses in the nature of rater judgments are generally considered to compromise the utility of workplace-based assessment (WBA). In order to gain insight into the underpinnings of rater behaviours, we investigated how raters form impressions of and make judgments on trainee performance. Using theoretical frameworks of social cognition and person perception, we explored raters’ implicit performance theories, use of task-specific performance schemas and the formation of person schemas during WBA. We used think-aloud procedures and verbal protocol analysis to investigate schema-based processing by experienced (N = 18) and inexperienced (N = 16) raters (supervisor-raters in general practice residency training). Qualitative data analysis was used to explore schema content and usage. We quantitatively assessed rater idiosyncrasy in the use of performance schemas and we investigated effects of rater expertise on the use of (task-specific) performance schemas. Raters used different schemas in judging trainee performance. We developed a normative performance theory comprising seventeen inter-related performance dimensions. Levels of rater idiosyncrasy were substantial and unrelated to rater expertise. Experienced raters made significantly more use of task-specific performance schemas compared to inexperienced raters, suggesting more differentiated performance schemas in experienced raters. Most raters started to develop person schemas the moment they began to observe trainee performance. The findings further our understanding of processes underpinning judgment and decision making in WBA. Raters make and justify judgments based on personal theories and performance constructs. Raters’ information processing seems to be affected by differences in rater expertise. The results of this study can help to improve rater training, the design of assessment instruments and decision making in WBA.

Keywords

Clinical education Cognition-based assessment models Competence assessment Performance assessment Professional judgment Rater judgments Rating process Workplace-based assessment