Advances in Health Sciences Education

, Volume 22, Issue 4, pp 969–983 | Cite as

The influence of first impressions on subsequent ratings within an OSCE station

  • Timothy J. Wood
  • James Chan
  • Susan Humphrey-Murto
  • Debra Pugh
  • Claire Touchie
Article

Abstract

Competency-based assessment is placing increasing emphasis on the direct observation of learners. For this process to produce valid results, it is important that raters provide quality judgments that are accurate. Unfortunately, the quality of these judgments is variable and the roles of factors that influence the accuracy of those judgments are not clearly understood. One such factor is first impressions: that is, judgments about people we do not know, made quickly and based on very little information. This study explores the influence of first impressions in an OSCE. Specifically, the purpose is to begin to examine the accuracy of a first impression and its influence on subsequent ratings. We created six videotapes of history-taking performance. Each video was scripted from a real performance by six examinee residents within a single OSCE station. Each performance was re-enacted with six different actors playing the role of the examinees and one actor playing the role of the patient and videotaped. A total of 23 raters (i.e., physician examiners) reviewed each video and were asked to make a global judgment of the examinee’s clinical abilities after 60 s (First Impression GR) by providing a rating on a six-point global rating scale and then to rate their confidence in the accuracy of that judgment by providing a rating on a five-point rating scale (Confidence GR). After making these ratings, raters then watched the remainder of the examinee’s performance and made another global rating of performance (Final GR) before moving on to the next video. First impression ratings of ability varied across examinees and were moderately correlated to expert ratings (r = .59, 95% CI [−.13, .90]). There were significant differences in mean ratings for three examinees. Correlations ranged from .05 to .56 but were only significant for three examinees. Rater confidence in their first impression was not related to the likelihood of a rater changing their rating between the first impression and a subsequent rating. The findings suggest that first impressions could play a role in explaining variability in judgments, but their importance was determined by the videotaped performance of the examinees. More work is needed to clarify conditions that support or discourage the use of first impressions.

Keywords

Rater cognition First impression OSCE Rater-based assessment 

References

  1. AERA, APA, & NCME. (2014). Standards for educational and psychological testing (pp. 11–31). Washington, DC: American Educational Research Association.Google Scholar
  2. Ambady, N. (2010). The perils of pondering: Intuition and thin slice judgments. Psychological Inquiry, 21(4), 271–278.CrossRefGoogle Scholar
  3. Ambady, N., Bernieri, F., & Richeson, J. (2000). Toward a histology of social behavior: Judgmental accuracy from thin slices of the behavioral stream. Advances in Experimental Social Psychology, 32, 201–271.CrossRefGoogle Scholar
  4. Ames, D. R., Kammrath, L. K., Suppes, A., & Bolger, N. (2010). Not so fast: The (not-quite-complete) dissociation between accuracy and confidence in thin-slice impressions. Personality and Social Psychology Bulletin, 36(2), 264–277.CrossRefGoogle Scholar
  5. Barrick, M. R., Swider, B. W., & Stewart, G. L. (2010). Initial evaluations in the interview: Relationships with subsequent interviewer evaluations and employment offers. The Journal of Applied Psychology, 95(6), 1163–1172.CrossRefGoogle Scholar
  6. Borkenau, P., & Liebler, A. (1992). Trait inferences: Sources of validity at zero acquaintance. Journal of Personality and Social Psychology, 62(4), 645–657.CrossRefGoogle Scholar
  7. Bown, M. H., Regehr, G., & Reznick, R. (1996). The effect of early performance on examiners’ marking patterns during an oral examination. Academic Medicine, 71(1), s73–s75.CrossRefGoogle Scholar
  8. Burdick, M. P., & Schoffstall, J. (1995). Observation of emergency medicine residents at the bedside: How often does it happen. Academic Emergency Medicine, 2(10), 909–913.CrossRefGoogle Scholar
  9. Carney, D., Colvin, C., & Hall, J. (2007). A thin slice perspective on the accuracy of first impressions. Journal of Research in Personality, 41(5), 1054–1072.CrossRefGoogle Scholar
  10. Cooper, W. H. (1981). Ubiquitous halo. Psychological Bulletin, 90(2), 218–244.CrossRefGoogle Scholar
  11. Croskerry, P. (2009). Clinical cognition and diagnostic error: Applications of a dual process model of reasoning. Advances in Health Sciences Education, 14, 27–35.CrossRefGoogle Scholar
  12. Dougherty, T. W., Turban, D. B., & Callender, J. C. (1994). Confirming first impressions in the employment interview: A field study of interview behaviour. Journal of Applied Psychology, 5(5), 659–665.CrossRefGoogle Scholar
  13. Evans, J. S. B. T. (2008). Dual-processing accounts of reasoning, judgment, and social cognition. Annual Review of Psychology, 59, 255–278.CrossRefGoogle Scholar
  14. Funder, D. C. (1987). Errors and mistakes: Evaluating the accuracy of social judgment. Psychological Bulletin, 101(1), 75–90.CrossRefGoogle Scholar
  15. Govaerts, M. J. B., Schuwirth, L. W. T., Van der Vleuten, C. P. M., & Muijtjens, A. M. M. (2011). Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education, 16(2), 151–165.CrossRefGoogle Scholar
  16. Harris, M., & Garris, C. (2008). You never get a second chance to make a first impression. In N. Ambady & J. Skowronski (Eds.), First impressions (pp. 147–168). New York: Guilford Press.Google Scholar
  17. Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., & Frank, J. R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676–682.CrossRefGoogle Scholar
  18. Kahneman, D. (2011). Thinking, fast and slow. Canada: Doubleday.Google Scholar
  19. Lopes, L. L. (1991). The rhetoric of irrationality. Theory and Psychology, 1(1), 65–82.CrossRefGoogle Scholar
  20. Macan, T. H., & Dipboye, R. L. (1990). The relationship of interviewer’s preinterview impressions to selection and recruitment outcomes. Personnel Psychology, 43(4), 745–768.CrossRefGoogle Scholar
  21. Martin, J. A., Reznick, R., Rothman, A., Tamblyn, R., & Regehr, G. (1996). Who should rate candidates in an objective structured clinical examination? Academic Emergency Medicine, 71(2), 170–175.Google Scholar
  22. Murphy, K. R., Jako, R. A., & Anhalt, R. L. (1993). Nature and consequences of halo error: A critical analysis. Journal of Applied Psychology, 78(2), 218–225.CrossRefGoogle Scholar
  23. Norman, G. R. (2009). Dual processing and diagnostic errors. Advances in Health Sciences Education, 14(1), 37–49.CrossRefGoogle Scholar
  24. Pelgrim, E. A., Kramer, A. W., Mokkink, H. G., & Van der Vleuten, C. P. M. (2012). The process of feedback in workplace-based assessment: Organisation, delivery, continuity. Medical Education, 46(6), 604–612.CrossRefGoogle Scholar
  25. Rannelli, L., Coderre, S., Paget, M., Woloschuk, W., Wright, B., & Mclaughlin, K. (2014). How do medical students form impressions of the effectiveness of classroom teachers? Medical Education, 48(8), 831–837.CrossRefGoogle Scholar
  26. Smith, H. J., Archer, D., & Costanzo, M. (1991). Just a hunch?: Accuracy and awareness in person perception. Journal of Nonverbal Behavior, 15(1), 3–18.CrossRefGoogle Scholar
  27. Touchie, C., Humphrey-Murto, S., Ainslie, M., Myers, K., & Wood, T. J. (2010). Two models of raters in a structured oral examination: Does it make a differnece? Advances in Health Sciences Education, 15(1), 97–108.CrossRefGoogle Scholar
  28. Williams, R. G., Dunnington, G. L., Mellinger, J. D., & Klamen, D. L. (2015). Placing constraints on the use of the ACGME milestones: A commentary on the limitations of global performance ratings. Academic Medicine, 90(4), 404–407.CrossRefGoogle Scholar
  29. Wood, T. J. (2014). Exploring the role of first impressions in rater-based assessments. Advances in Health Sciences Education, 19(3), 409–427.CrossRefGoogle Scholar
  30. Yaphe, J., & Street, S. (2003). How do examiners decide?: A qualitative study of the process of decision making in the oral examination component of the MRCGP examination. Medical Education, 37(9), 764–771.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2016

Authors and Affiliations

  • Timothy J. Wood
    • 1
  • James Chan
    • 2
  • Susan Humphrey-Murto
    • 2
  • Debra Pugh
    • 2
  • Claire Touchie
    • 2
  1. 1.Department of Innovation in Medical Education, Faculty of MedicineUniversity of OttawaOttawaCanada
  2. 2.Department of MedicineUniversity of OttawaOttawaCanada

Personalised recommendations