Exploring the role of first impressions in rater-based assessments
- 2.2k Downloads
Medical education relies heavily on assessment formats that require raters to assess the competence and skills of learners. Unfortunately, there are often inconsistencies and variability in the scores raters assign. To ensure the scores from these assessment tools have validity, it is important to understand the underlying cognitive processes that raters use when judging the abilities of their learners. The goal of this paper, therefore, is to contribute to a better understanding of the cognitive processes used by raters. Representative findings from the social judgment and decision making, cognitive psychology, and educational measurement literature will be used to enlighten the underpinnings of these rater-based assessments. Of particular interest is the impact judgments referred to as first impressions (or thin slices) have on rater-based assessments. These are judgments about people made very quickly and based on very little information. A narrative review will provide a synthesis of research in these three literatures (social judgment and decision making, educational psychology, and cognitive psychology) and will focus on the underlying cognitive processes, the accuracy and the impact of first impressions on rater-based assessments. The application of these findings to the types of rater-based assessments used in medical education will then be reviewed. Gaps in understanding will be identified and suggested directions for future research studies will be discussed.
KeywordsFirst impressions Rater-based assessment Rater-cognition
The author would like to thank Dr. Andrea Gingerich and Dr Sydney Smee for their helpful discussions that led to this project and Dr. Stan Hamstra, Dr. Susan Humprey-Murto, Kulamakan Mahan Kulasegaram, Sarah Lynch, Dr. Debra Pugh, Dr. Nikki Woods and the journal editor for their thoughtful review and suggestions regarding this manuscript.
- AERA, APA, & NCME. (1999). Standards for educational and psychological testing (pp. 9–24). Washington, DC: American Educational Research Association.Google Scholar
- Berendonk, C., Stalmeijer, R. E., & Schuwirth, L. W. T. (2013). Expertise in performance assessment: Assessors perspectives. Advances in Health Sciences Education: Theory and Practice. doi: 10.1007/s10459-012-9392-x
- Biesanz, J. C., Human, L. J., Paquin, A. C., Chan, M., Parisotto, K. L., Sarracino, J., et al. (2011). Do we know when our impressions of others are valid? Evidence for realistic accuracy awareness in first impressions of personality. Social Psychological and Personality Science, 2(5), 452–459.CrossRefGoogle Scholar
- Clauser, B. E., Margolis, M. J., & Swanson, D. B. (2008). Issues of validity and reliability for assessments in Medical Education. In E. S. Holmboe & R. E. Hawkins (Eds.), Practical guide to the evaluation of clinical competence (pp. 10–23). Philadelphia: Mosby Elsevier.Google Scholar
- Dipboye, R. L. (1982). Self-fulfilling prophecies in the selection-recruitment interview. The Academy of Management Review, 7(4), 579.Google Scholar
- Downing, S. M., & Haladyna, T. M. (2009). Validity and its threats. In S. M. Downing & R. Yudkowsky (Eds.), Assessment in health professions education (pp. 21–56). New York: Routledge.Google Scholar
- Fiske, S., & Neuberg, S. (1990). A continuum of impression formation, from category-based to individuating processes: Influences of information and motivation on attention and interpretation. In M. Zanna (Ed.), Advances in experimental social psychology (23rd ed., pp. 1–75). San Diego: Academic Press Inc.Google Scholar
- Govaerts, M. J. B., Van de Wiel, M. W. J., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2013). Workplace-based assessment: Raters’ performance theories and constructs. Advances in Health Sciences Education: Theory and Practice. doi: 10.1007/s10459-012-9376-x.
- Harris, M., & Garris, C. (2008). You never get a second chance to make a first impression. In N. Ambady & J. Skowronski (Eds.), First impressions (pp. 147–168). New York, NY: Guilford Press.Google Scholar
- Hawkins, R. E., & Boulet, J. R. (2008). Direct observation: Standardized patients. In E. S. Holmboe & R. E. Hawkins (Eds.), Evaluation of clinical competence (pp. 102–118). Philadelphia, PA: Mosby Elsevier.Google Scholar
- Jacoby, L., & Kelley, C. (1990). An episodic view of motivation: Unconscious influences of memory. In E. T. Higgins & R. M. Sorrentino (Eds.), Handbook of motivation and cognition: Foundations of social behavior (Vol. 2, pp. 451–480). New York, NY: Guilford Press.Google Scholar
- Kahneman, D. (2011). Thinking, fast and slow. Canada: Doubleday.Google Scholar
- Klein, G. (2009). Streetlights and shadows: Searching for the keys to adaptive decision making. Cambridge, MA: MIT Press.Google Scholar
- Monteiro, S. D., Sherbino J. D., Ilgen, J. S., Dore, K. L. Gaissmaier, W., Wood, T. J., et al. (unpublished manuscript). Diagnosing Fast and Slow: The Effect of Interruptions on Speeded and Reflective Clinical Reasoning. Google Scholar
- Norman, G. R., Sherbino, J., Dore, K. L., Wood, T. J. Ph. Young, M. E., Gaissmaier, W., et al. (in press). The etiology of diagnostic errors: A controlled trial of System 1 vs. System 2 reasoning. Academic Medicine. Google Scholar
- Norman, G., Young, M., & Brooks, L. (2007). Non-analytical models of clinical reasoning: The role of experience. Medical Education, 41, 1140–1145.Google Scholar
- Tavares, W., & Eva, K. W. (2013). Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education: Theory and Practice. doi: 10.1007/s10459-012-9370-3.
- Wigton, R. (1980). The effects of student personal characteristics on the evaluation of clinical performance. Journal of Medical Education, 55, 423–427.Google Scholar
- Wood, T. J. (2013). Mental workload as a tool for understanding dual processes in rater-based assessments. Advances in Health Sciences Education: Theory and Practice. doi: 10.1007/s10459-012-9396-6
- Yeates, P., O’Neill, P., Mann, K., & Eva, K. (2013). Seeing the same thing differently. Advances in Health Sciences Education: Theory and Practice. doi: 10.1007/s10459-012-9372-1.