Abstract
While checklists are often used to score standardized patient based clinical assessments, little research has focused on issues related to their development or the level of agreement with respect to the importance of specific items. Five physicians independently reviewed checklists from 11 simulation scenarios that were part of the former Educational Commission for Foreign Medical Graduate’s Clinical Skills Assessment and classified the clinical appropriateness of each of the checklist items. Approximately 78% of the original checklist items were judged to be needed, or indicated, given the presenting complaint and the purpose of the assessment. Rater agreement was relatively poor with pairwise associations (Kappa coefficient) ranging from 0.09 to 0.29. However, when only consensus indicated items were included, there was little change in examinee scores, including their reliability over encounters. Although most checklist items in this sample were judged to be appropriate, some could potentially be eliminated, thereby minimizing the scoring burden placed on the standardized patients. Periodic review of checklist items, concentrating on their clinical importance, is warranted.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adamo G. (2003). Simulated and standardized patients in OSCEs: achievements and challenges 1992–2003. Medical Teacher 25 262–270
Boulet J.R., McKinley D.W., Norcini J.J., Whelan G.P. (2002). Assessing the comparability of standardized patient and physician evaluations of clinical skills. Advances in Health Sciences Education: Theory and Practice 7:85–97
Boulet J.R., McKinley D.W., Whelan G.P., Hambleton R.K. (2003). Quality assurance methods for performance-based assessments. Advances in Health Sciences Education: Theory and Practice 8:27–47
De Champlain A.F., Macmillan M.K., King A.M., Klass D.J., Margolis M.J. (1999). Assessing the impacts of intra-site and inter-site checklist recording discrepancies on the reliability of scores obtained in a nationally administered standardized patient examination. Academic Medicine 74:S52–S54
Federation of State Medical Boards & National Board of Medical Examiners (2003). 2004 USMLE Step 2 CS Content Description and General Information Booklet Philadelphia: FSMB and NBME
Gorter S., Rethans J.J., Scherpbier A., Heijde D., Houben H., Vleuten C., et al. (2000). Developing case-specific checklists for standardized-patient-based assessments in internal medicine: A review of the literature. Academic Medicine 75:1130–1137
Landis J.R., Koch G.G. (1977). The measurement of observer agreement for categorical data. Biometrics 33:159–173
Newble D. (2004). Techniques for measuring clinical competence: objective structured clinical examinations. Medical Education 38:199–203
Norcini J., Boulet J. (2003). Methodological issues in the use of standardized patients for assessment. Teaching and Learning in Medicine 15:293–297
van der Vleuten C.P., Norman G.R., De Graaff E. (1991). Pitfalls in the pursuit of objectivity: issues of reliability. Medical Education 25:110–118
Whelan G.P., Boulet J.R., McKinley D.W., Norcini J.J., van Zanten M., Hambleton R.K. et al. (2005). Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA). Medical Teacher 27:200–206
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Boulet, J.R., van Zanten, M., de Champlain, A. et al. Checklist Content on a Standardized Patient Assessment: An Ex Post Facto Review. Adv in Health Sci Educ 13, 59–69 (2008). https://doi.org/10.1007/s10459-006-9024-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-006-9024-4