Inflated Clinical Evaluations: a Comparison of Faculty-Selected and Mathematically Calculated Overall Evaluations Based on Behaviorally Anchored Assessment Data
This retrospective study compared faculty-selected evaluation scores with those mathematically calculated from behaviorally anchored assessments.
Data from 1036 psychiatry clerkship clinical evaluations (2012–2015) was reviewed. These clinical evaluations required faculty to assess clinical performance using 14 behaviorally anchored questions followed by a faculty-selected overall evaluation. An explicit rubric was included in the overall evaluation to assist the faculty in interpreting their 14 assessment responses. Using the same rubric, mathematically calculated evaluations of the same assessment responses were generated and compared to the faculty-selected evaluations.
Comparison of faculty-selected to mathematically calculated evaluations revealed that while the two methods were reliably correlated (Cohen’s kappa = 0.314, Pearson’s coefficient = 0.658, p < 0.001), there was a notable difference in the results (t = 24.5, p < 0.0001). The average faculty-selected evaluation was 1.58 (SD = 0.61) with a mode of “1” or “outstanding,” while the mathematically calculated evaluation had an average of 2.10 (SD = 0.90) with a mode of “3” or “satisfactory.” 51.0% of the faculty-selected evaluations matched the mathematically calculated results: 46.1% were higher and 2.9% were lower.
Clerkship clinical evaluation forms that require faculty to make an overall evaluation generate results that are significantly higher than what would have been assigned solely using behavioral anchored assessment questions. Focusing faculty attention on assessing specific behaviors rather than overall evaluations may reduce this inflation and improve validity. Clerkships may want to consider removing overall evaluation questions from their clinical evaluation tools.
KeywordsCompetency Evaluation Assessment UME Medical student
We wish to thank the faculty and students of the USUHS Psychiatry Clerkship.
Compliance with Ethical Standards
This research protocol was reviewed and declared exempt by the USUHS institutional review board (IRB) in accordance with all applicable Federal regulations governing the protection of animals in research. (PSY-88-9072).
The opinions and assertions expressed herein are those of the author(s) and do not necessarily reflect the official policy or position of the Uniformed Services University or the Department of Defense.
None of the authors have a financial relationship with any entity producing, marketing, re-selling, or distributing healthcare goods or services consumed by, or used on, patients.
This work was prepared by military employees of the US Government as part of the individual’s official duties and therefore is in the public domain and does not possess copyright protection (public domain information may be freely distributed and copied; however, as a courtesy it is requested that the Uniformed Services University and the author be given an appropriate acknowledgement).
- 2.Cooke M, Irby DM, O’Brien BC. Educating physicians: a call for reform of medical school and residency. John Wiley & Sons; 2010.Google Scholar
- 5.Pangaro LN, McGaghie WC, editors. Handbook on medical student evaluation and assessment. Gegensatz Press; 2015.Google Scholar
- 7.Swaffield S. The misrepresentation of Assessment for Learning – and the woeful waste of a wonderful opportunity. In: The 20th Annual Conference of the Association for Achievement and Improvement through Assessment (AAIA), Bournemouth. 2009. https://www.aaia.org.uk/content/uploads/2010/07/The-Misrepresentation-of-Assessment-for-Learning.pdf. Accessed 6 Jun 2018.
- 16.Battistone MJ, Pendleton B, Milne C, Battistone ML, Sande MA, Hemmer PA, et al. Global descriptive evaluations are more responsive than global numeric ratings in detecting Students’ progress during the inpatient portion of an internal medicine clerkship. Acad Med. 2001;76(10):S105–7.CrossRefPubMedGoogle Scholar
- 20.Holmboe ES, Edgar L, Hamstra S. The milestones guidebook. Accreditation Council for Graduate Medical Education, 2016.Google Scholar