Advances in Health Sciences Education

, Volume 7, Issue 2, pp 85–97 | Cite as

Assessing the Comparability of Standardized Patient and Physician Evaluations of Clinical Skills

  • John R. Boulet
  • Danette W. McKinley
  • John J. Norcini
  • Gerald P. Whelan
Article

Abstract

Accumulating evidence to defend decisions based on scores from evaluations is an ongoing process. The purpose of this investigation was to gather additional data to support the validity of inferences made from scores on the Educational Commission for Foreign Medical Graduates' Clinical Skills Assessment (CSA®). This was accomplished by contrasting CSA candidate scores, and pass/fail decisions, with those obtained from the American Board of Internal Medicine's Mini-CEX (Clinical Evaluation Exercise). Data gathering performance based on the number of unweighted history taking and physical examination checklist items adequately predicted the global ratings provided by physician observers. CSA ratings of doctor-patient communication skills correlated with mini-CEX ratings of like constructs, indicating that physician observers, using mini-CEX rating scales, are able to make realistic assessments of interpersonal skills. These results provide evidence of the convergent validity of CSA scores.

clinical skills certification tests physician judges standardized patients validity 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. American Educational Research Association, American Psychological Association and National Council on Measurement in Education (1999). Standards for Educational and Psychological Testing, p. 9. Washington, DC: American Educational Research Association.Google Scholar
  2. Boulet, J.R., Friedman Ben-David, M., Ziv, A., Burdick, W.P., Curtis, M., Peitzman, S. & Gary, N.E. (1998). Using standardized patients to assess the interpersonal skills of physicians. Academic Medicine 73: S94–S96.CrossRefGoogle Scholar
  3. Boulet, J.R., Friedman Ben-David, M., Ziv, A., Burdick, W.P. & Gary, N.E. (2000). The use of holistic scoring for post-encounter written exercises. In D. Melnick (ed.), Proceedings of the Eighth Ottawa Conference of Medical Education and Assessment, pp. 254–260. Philadelphia: National Board of Medical Examiners.Google Scholar
  4. Educational Commission for Foreign Medical Graduates (1999). Clinical Skills Assessment (CSA) Candidate Orientation Manual. Philadelphia: Educational Commission for Foreign Medical Graduates (ECFMG).Google Scholar
  5. Gray, J.D. (1996). Global rating scales in residency education. Academic Medicine 71: S55–S63.CrossRefGoogle Scholar
  6. Hodges, B., Regehr, G., McNaughton, N., Tiberius, R. & Hanson, M. (1999). OSCE checklists do not capture increasing levels of expertise. Academic Medicine 74: 1129–1134.CrossRefGoogle Scholar
  7. Holmboe, E.S. & Hawkins, R.E. (1998). Methods for evaluating the clinical competence of residents in internal medicine: A review. Annals of Internal Medicine 129: 42–48.Google Scholar
  8. Norcini, J.J., Blank, L.L., Arnold, G.K. & Kimball, H.R. (1995). The mini-CEX (clinical evaluation exercise): A preliminary investigation. Annals of Internal Medicine 123: 795–799.Google Scholar
  9. Norcini, J.J., Blank, L.L., Arnold, G.K. & Kimball, H.R. (1997). Examiner differences in the mini-CEX. Advances in Health Sciences Education 2: 27–33.CrossRefGoogle Scholar
  10. Regehr, G., Freeman, R., Robb, A., Missiha, N. & Heisey, R. (1999). OSCE performance evaluations made by standardized patients: Comparing checklist and global rating scores. AcademicMedicine 74: S135–S137.Google Scholar
  11. Reznick, R., Blackmore, D., Dauphinee, W.D., Rothman, A. & Smee, S. (1996). Large-scale highstakes testing with an OSCE: Report from the Medical College of Canada. Academic Medicine 71: S19–S21.CrossRefGoogle Scholar
  12. Reznick, R., Regehr, G., Yee, G., Rothman, A., Blackmore, D. & Dauphinee, D. (1998). Processrating forms versus task-specific checklists in an OSCE for medical licensure. Academic Medicine 73: S97–S99.CrossRefGoogle Scholar
  13. Swanson, D.B., Norman, G.R. & Linn, R.L. (1995). Performance-based assessment: Lessons from the health professions. Educational Researcher 24: 5–11.CrossRefGoogle Scholar
  14. Ziv, A., Boulet, J.R., Burdick, W.P., Friedman Ben-David, M. & Gary, N.E. (2000). The use of national medical care surveys to develop and validate test content for standardized patient examinations. In D. Melnick (ed.), Proceedings of the Eighth Ottawa Conference on Medical Education and Assessment, pp. 99–105. Philadelphia: National Board of Medical Examiners.Google Scholar
  15. Ziv, A., Boulet, J.R., Friedman Ben-David, M., Curtis, M., Burdick, W.P., Peitzman, S. & Gary, N.E. (2000). A holistic and behaviorally anchored measure of physician communication skills: Issues of rater consistency. In D. Melnick (ed.), Proceedings of the Eighth Ottawa Conference on Medical Education and Assessment, pp. 247–253. Philadelphia: National Board of Medical Examiners.Google Scholar

Copyright information

© Kluwer Academic Publishers 2002

Authors and Affiliations

  • John R. Boulet
    • 1
  • Danette W. McKinley
    • 1
  • John J. Norcini
    • 2
  • Gerald P. Whelan
    • 1
  1. 1.Educational Commission for Foreign Medical GraduatesPhiladelphiaU.S.A.
  2. 2.American Board of Internal MedicineU.S.A

Personalised recommendations