Advances in Health Sciences Education

, Volume 9, Issue 1, pp 47–60

Assessing the Written Communication Skills of Medical School Graduates

  • John R. Boulet
  • Thomas A. Rebbecchi
  • Elizabeth C. Denton
  • Danette W. McKinley
  • Gerald P. Whelan


The ECFMG® Clinical Skills Assessment (CSA®)was developed to evaluate whether graduates of international medical schools (IMGs) are ready to enter graduate training programs in the United States. The patient note (PN) exercise, conducted after a 15-minute interview with a standardized patient (SP), is specifically used to assess a candidate's ability to summarize and synthesize the data collected. On a yearly basis, approximately 75,000 patient notes are reviewed and scored by physician raters. Recent changes to the PN scoring rubric, combined with enhancements to quality assurance procedures, mandate that additional evidence be provided to support the intended use of PN scores. The purpose of this study was to further investigate the psychometric adequacy of PN scores. Generalizability analyses suggest that while variability in PN rating scan be attributed to the choice of rater, candidate scores are reproducible over the 10-encounter CSA. The relationship of PN scores with other related ability measures and select candidate characteristics provides additional evidence to support the validity of the written exercise.

certification tests reliability standardized patient validity written communication 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. American Educational Research Association, American Psychological Association, National Council on Measurement in Education (1999). Standards for Educational and Psychological Testing. Washington, DC: Author.Google Scholar
  2. Australian Council for Educational Research (1998). ACER ConQuest: Generalized Item Response Modelling Software. Melbourne, Australia: Author.Google Scholar
  3. Ayers, W.R. & Boulet, J.R. (2001). Establishing the validity of test score inferences: Performance of 4th-year U.S. medical students on the ECFMG Clinical Skills Assessment. Teaching and Learning in Medicine 13(4): 214-220.CrossRefGoogle Scholar
  4. Boulet, J.R., Ben David, M.F., Ziv, A., Burdick,W.P., Curtis,M. & Peitzman, S. et al. (1998a). Using standardized patients to assess the interpersonal skills of physicians. Academic Medicine 73(10 Suppl.): 94-96.CrossRefGoogle Scholar
  5. Boulet, J.R., Friedman Ben-David, M., Hambleton, R.K., Burdick, W.P., Ziv, A. & Gary, N.E. (1998b). An investigation of the sources of measurement error in the post-encounter written scores from standardized patient examinations. Advances in Health Sciences Education: Theory and Practice 3: 89-100.CrossRefGoogle Scholar
  6. Boulet, J.R., Friedman Ben-David, M., Ziv, A., Burdick, W.P. & Gary, N.E. (2000). The use of holistic scoring for post-encounter written exercises. In D. Melnick (ed.), Proceedings of the Eighth Ottawa Conference on Medical Education and Assessment (pp. 254-260). Philadelphia, PA: National Board of Medical Examiners.Google Scholar
  7. Boulet, J.R., McKinley, D.W., Norcini, J.J. & Whelan, G.P. (2002). Assessing the comparability of standardized patient and physician evaluations of clinical skills. Advances in Health Sciences Education: Theory and Practice 7: 85-97.CrossRefGoogle Scholar
  8. Boulet, J.R., van Zanten, M., McKinley, D.W. & Gary, N.E. (2001). Evaluating the spoken English proficiency of graduates of foreign medical schools. Medical Education 35(8): 767-773.CrossRefGoogle Scholar
  9. Brennan, R.L. (2001). Generalizability Theory. New York: Springer-Verlag.Google Scholar
  10. Cradock, J., Young, A.S. & Sullivan, G. (2001). The accuracy of medical record documentation in schizophrenia. Journal of Behavioral Health Services & Research 28(4): 456-465.CrossRefGoogle Scholar
  11. Crossley, G.M., Howe, A., Newble, D., Jolly, B. & Davies, H.A. (2001). Sheffield Assessment Instrument for Letters (SAIL): Performance assessment using outpatient letters. Medical Education 35(12): 1115-1124.CrossRefGoogle Scholar
  12. Friedman Ben-David, M.F., Boulet, J.R., Burdick, W.P., Ziv, A., Hambleton, R.K. & Gary, N.E. (1997). Issues of validity and reliability concerning who scores the post-encounter patientprogress note. Academic Medicine 72(10 Suppl. 1): 79-81.CrossRefGoogle Scholar
  13. Friedman Ben-David, M.F., Klass, D.J., Boulet, J., De Champlain, A., King, A.M. & Pohl, H.S. et al. (1999). The performance of foreign medical graduates on the National Board of Medical Examiners (NBME) standardized patient examination prototype: A collaborative study of the NBME and the Educational Commission for Foreign Medical Graduates (ECFMG). Medical Education 33(6): 439-446.CrossRefGoogle Scholar
  14. Grace-Farfaglia, P. & Rosow, P. (1995). Automating clinical dietetics documentation Journal of the American Dietetic Association 95(6): 687-690.CrossRefGoogle Scholar
  15. Hallock, J.A. ECFMG and the Challenges Facing International Medical Graduates (2002). Reporter 11(8): 2-3. Washington, DC: Association of American Medical Colleges.Google Scholar
  16. Harden, R.M. & Gleeson, F.A. (1979). Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education 13(1): 41-54.Google Scholar
  17. Howell, J., Chisholm, C., Clark, A. & Spillane, L. (2000). Emergency medicine resident documentation: Results of the 1999 American Board of Emergency Medicine in-training examination survey. Academic Emergency Medicine 7(10): 1135-1138.CrossRefGoogle Scholar
  18. Larimore, W.L. & Jordan, E.V. (1995). SOAP to SNOCAMP: improving the medical record format. Journal of Family Practice 41(4): 393-398.Google Scholar
  19. Medical Council of Canada (2002). Qualifying Examination Part II, Information Pamphlet. Ottawa, Canada: Author.Google Scholar
  20. Prentice, D.A. & Miller, D.T. (1998). When small effects are impressive. In A.E. Kazdin (ed.), Methodological Issues and Strategies in Clinical Research (pp. 163-173). Washington, DC: American Psychological Association.Google Scholar
  21. Regehr, G., MacRae, H., Reznick, R.K. & Szalay, D. (1998). Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Academic Medicine 73(9): 993-997.CrossRefGoogle Scholar
  22. Rollnick, S., Kinnersley, P. & Butler, C. (2002). Context-bound communication skills training: Development of a new method. Medical Education 36(4): 377-383.CrossRefGoogle Scholar
  23. SAS Institute, Inc. (1989). SAS/STAT User's Guide: Version 6 (4th ed.), Cary, NC: Author.Google Scholar
  24. Shapiro, J. (1999). Correlates of family-oriented physician communications. Family Practice 16(3): 294-300.CrossRefGoogle Scholar
  25. Slater, S.C. & Boulet, J.R. (2001). Predicting holistic ratings of written performance assessments from analytic scoring. Advances in Health Sciences Education: Theory and Practice 6(2): 103-119.CrossRefGoogle Scholar
  26. Sleszynski, S.L., Glonek, T. & Kuchera, W.A. (1999). Standardized medical record: A new outpatient osteopathic SOAP note form: Validation of a standardized office form against physician's progress notes. Journal of the American Osteopathic Association 99(10): 516-529.Google Scholar
  27. Swanson, D.B. & Norcini, J.J. (1989). Factors influencing reproducibility of tests using standardized patients. Teaching and Learning in Medicine 1(3): 158-166.CrossRefGoogle Scholar
  28. van Dalen, J., Kerkhofs, E., Verwijnen, G.M., Knippenberg-Van Den Berg, B.W., van Den Hout, H.A., Scherpbier, A.J. et al. (2002). Predicting communication skills with a paper-and-pencil test. Medical Education 36(2): 148-153.CrossRefGoogle Scholar
  29. van der Vleuten, C.P., Norman, G.R. & De Graaff, E. (1991). Pitfalls in the pursuit of objectivity: Issues of reliability. Medical Education 25(2): 110-118.CrossRefGoogle Scholar
  30. Weber, D.O. (2002). Charting a course toward legible medical records: Perfect paperwork can mean financial savings, better patient care. Physician Executive 28(1): 8-13.Google Scholar
  31. Williams, S., Weinman, J. & Dale, J. (1998). Doctor-patient communication and patient satisfaction: A review. Family Practice 15(5): 480-492.CrossRefGoogle Scholar

Copyright information

© Kluwer Academic Publishers 2004

Authors and Affiliations

  • John R. Boulet
    • 1
  • Thomas A. Rebbecchi
    • 1
  • Elizabeth C. Denton
    • 1
  • Danette W. McKinley
    • 1
  • Gerald P. Whelan
    • 1
  1. 1.Research and EvaluationEducational Commission for Foreign Medical Graduates (ECFMG)PhiladelphiaUSA

Personalised recommendations