Advertisement

Advances in Health Sciences Education

, Volume 13, Issue 1, pp 59–69 | Cite as

Checklist Content on a Standardized Patient Assessment: An Ex Post Facto Review

  • John R. BouletEmail author
  • Marta van  Zanten
  • André de Champlain
  • Richard E. Hawkins
  • Steven J. Peitzman
Article

Abstract

While checklists are often used to score standardized patient based clinical assessments, little research has focused on issues related to their development or the level of agreement with respect to the importance of specific items. Five physicians independently reviewed checklists from 11 simulation scenarios that were part of the former Educational Commission for Foreign Medical Graduate’s Clinical Skills Assessment and classified the clinical appropriateness of each of the checklist items. Approximately 78% of the original checklist items were judged to be needed, or indicated, given the presenting complaint and the purpose of the assessment. Rater agreement was relatively poor with pairwise associations (Kappa coefficient) ranging from 0.09 to 0.29. However, when only consensus indicated items were included, there was little change in examinee scores, including their reliability over encounters. Although most checklist items in this sample were judged to be appropriate, some could potentially be eliminated, thereby minimizing the scoring burden placed on the standardized patients. Periodic review of checklist items, concentrating on their clinical importance, is warranted.

Keywords

checklist inter-rater agreement reliability standardized patient validity 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adamo G. (2003). Simulated and standardized patients in OSCEs: achievements and challenges 1992–2003. Medical Teacher 25 262–270CrossRefGoogle Scholar
  2. Boulet J.R., McKinley D.W., Norcini J.J., Whelan G.P. (2002). Assessing the comparability of standardized patient and physician evaluations of clinical skills. Advances in Health Sciences Education: Theory and Practice 7:85–97CrossRefGoogle Scholar
  3. Boulet J.R., McKinley D.W., Whelan G.P., Hambleton R.K. (2003). Quality assurance methods for performance-based assessments. Advances in Health Sciences Education: Theory and Practice 8:27–47CrossRefGoogle Scholar
  4. De Champlain A.F., Macmillan M.K., King A.M., Klass D.J., Margolis M.J. (1999). Assessing the impacts of intra-site and inter-site checklist recording discrepancies on the reliability of scores obtained in a nationally administered standardized patient examination. Academic Medicine 74:S52–S54CrossRefGoogle Scholar
  5. Federation of State Medical Boards & National Board of Medical Examiners (2003). 2004 USMLE Step 2 CS Content Description and General Information Booklet Philadelphia: FSMB and NBMEGoogle Scholar
  6. Gorter S., Rethans J.J., Scherpbier A., Heijde D., Houben H., Vleuten C., et al. (2000). Developing case-specific checklists for standardized-patient-based assessments in internal medicine: A review of the literature. Academic Medicine 75:1130–1137CrossRefGoogle Scholar
  7. Landis J.R., Koch G.G. (1977). The measurement of observer agreement for categorical data. Biometrics 33:159–173CrossRefGoogle Scholar
  8. Newble D. (2004). Techniques for measuring clinical competence: objective structured clinical examinations. Medical Education 38:199–203CrossRefGoogle Scholar
  9. Norcini J., Boulet J. (2003). Methodological issues in the use of standardized patients for assessment. Teaching and Learning in Medicine 15:293–297CrossRefGoogle Scholar
  10. van der Vleuten C.P., Norman G.R., De Graaff E. (1991). Pitfalls in the pursuit of objectivity: issues of reliability. Medical Education 25:110–118CrossRefGoogle Scholar
  11. Whelan G.P., Boulet J.R., McKinley D.W., Norcini J.J., van Zanten M., Hambleton R.K. et al. (2005). Scoring standardized patient examinations: lessons learned from the development and administration of the ECFMG Clinical Skills Assessment (CSA). Medical Teacher 27:200–206CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, Inc. 2006

Authors and Affiliations

  • John R. Boulet
    • 1
    Email author
  • Marta van  Zanten
    • 1
  • André de Champlain
    • 2
  • Richard E. Hawkins
    • 2
  • Steven J. Peitzman
    • 1
  1. 1.Educational Commission for Foreign Medical Graduates (ECFMG®)PhiladelphiaUSA
  2. 2.National Board of Medical Examiners (NBME®)PhiladelphiaUSA

Personalised recommendations