Skip to main content
SpringerLink
Log in

Multi-source evaluation of interpersonal and communication skills of family medicine residents

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

There is a lack of information on the use of multi-source evaluation to assess trainees’ interpersonal and communication skills in Oriental settings. This study is conducted to assess the reliability and applicability of assessing the interpersonal and communication skills of family medicine residents by patients, peer residents, nurses, and teaching staffs and to compare the ratings with the objective structured clinical examination (OSCE). Our results revealed instruments used by staffs, peers, nurses, and self-evaluation have good internal consistency reliability (α > 0.90), except for the behavioral checklist (α = 0.57). Staffs’, peers’, and nurses’ evaluations were highly correlated with one another (r = 0.722 for staff- and peer-rating, r = 0.734 for staff- and nurse-rating, r = 0.634 for peer- and nurse-rating). However, residents’ self-rating and patients-rating were not correlated to ratings by any other raters. OSCE evaluation was correlated to peer-rating (r = 0.533) and staff-rating (r = 0.642), but not correlated to self- or patient-rating. The generalizability study revealed the major sources of variance came from the types of rater and the interaction of residents and types of rater. This study found self-rating and patient-rating were not consistent with other sources of rating on residents’ interpersonal and communication skills. Whether variations among different types of rater in a multi-source evaluation should be regarded as measurement errors or complementary information is worth further study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Becker, M. H. (1985). Patient adherence to prescribed therapies. Medical Care, 23(5), 539–555.

    Article  Google Scholar 

  • Brennan, R. L. (2001). Generalizability theory. New York: Springer.

    Google Scholar 

  • Brown, J. B., Boles, M., Mullooly, J. P., & Levinson, W. (1999). Effect of clinician communication skills training on patient satisfaction. A randomized, controlled trial. Annal of Internal Medicine, 131(11), 822–829.

    Google Scholar 

  • Butterfield, P. S., & Mazzaferri, E. L. (1991). A new rating form for use by nurses in assessing residents’ humanistic behavior. Journal of General Internal Medicine, 6(2), 155–161.

    Article  Google Scholar 

  • Carraccio, C., & Englander, R. (2000). The objective structured clinical examination: A step in the direction of competency-based evaluation. Archives of Pediatrics Adolescent Medicine, 154(7), 736–741.

    Google Scholar 

  • Donnelly, M. B., Sloan, D., Plymale, M., & Schwartz, R. (2000). Assessment of residents’ interpersonal skills by faculty proctors and standardized patients: a psychometric analysis. Academic Medicine, 75(10 Suppl), S93–S95.

    Article  Google Scholar 

  • Duffy, F. D., Gordon, G. H., Whelan, G., Cole-Kelly, K., Frankel, R., Buffone, N., et al. (2004). Assessing competence in communication and interpersonal skills: The Kalamazoo II report. Academic Medicine, 79(6), 495–507.

    Article  Google Scholar 

  • Eva, K. W., & Regehr, G. (2005). Self-assessment in the health professions: A reformulation and research agenda. Academic Medicine, 80(10 Suppl), S46–S54.

    Article  Google Scholar 

  • Eva, K. W., & Regehr, G. (2007). Knowing when to look it up: A new conception of self-assessment ability. Academic Medicine, 82(10 suppl), S81–S84.

    Article  Google Scholar 

  • Fidler, H., Lockyer, J. M., Toews, J., & Violato, C. (1999). Changing physicians’ practices: The effect of individual feedback. Academic Medicine, 74(6), 702–714.

    Article  Google Scholar 

  • Frank, J. R., & Danoff, D. (2007). The CanMEDS initiative: Implementing an outcomes-based framework of physician competencies. Medical Teacher, 29(7), 642–647.

    Article  Google Scholar 

  • Gordon, M. J. (1991). A review of the validity and accuracy of self-assessments in health professions training. Academic Medicine, 66(12), 762–769.

    Article  Google Scholar 

  • Greiner, A. C., & Knebel, E. (Eds.). (2003). Health professions education: A bridge to quality. Washington, DC: The National Academies Press.

    Google Scholar 

  • Guiton, G., Hodgson, C. S., Delandshere, G., & Wilkerson, L. (2004). Communication skills in standardized-patient assessment of final-year medical students: A psychometric study. Advances in Health Sciences Education, 9(3), 179–187.

    Article  Google Scholar 

  • Hagedoorn, M., Uijl, S. G., Van Sonderen, E., Ranchor, A. V., Grol, B. M., Otter, R., et al. (2003). Structure and reliability of Ware’s Patient Satisfaction Questionnaire III: patients’ satisfaction with oncological care in the Netherlands. Medical Care, 41(2), 254–263.

    Google Scholar 

  • Hall, J. A., Irish, J. T., Roter, D. L., Ehrlich, C. M., & Miller, L. H. (1994). Satisfaction, gender, and communication in medical visits. Medical Care, 32(12), 1216–1231.

    Article  Google Scholar 

  • Hall, W., Violato, C., Lewkonia, R., Lockyer, J., Fidler, H., Toews, J., et al. (1999). Assessment of physician performance in Alberta: The physician achievement review. Canadian Medical Association Journal, 161(1), 52–57.

    Google Scholar 

  • Heard, J. K., Allen, R. M., Clardy, J., Heard, J. K., Allen, R. M., & Clardy, J. (2002). Assessing the needs of residency program directors to meet the ACGME general competencies. Academic Medicine, 77(7), 750.

    Article  Google Scholar 

  • Hodges, B. (2003). Validity and the OSCE. Medical Teacher, 25(3), 250–254.

    Article  Google Scholar 

  • Joshi, R., Ling, F. W., Jaeger, J., Joshi, R., Ling, F. W., & Jaeger, J. (2004). Assessment of a 360-degree instrument to evaluate residents’ competency in interpersonal and communication skills. Academic Medicine, 79(5), 458–463.

    Article  Google Scholar 

  • Kurtz, S. M. (2002). Doctor-patient communication: Principles and practices. Canadian Journal of Neurological Sciences, 29(Suppl 2), S23–S29.

    Google Scholar 

  • Kurtz, S. M., & Silverman, J. D. (1996). The Calgary-Cambridge referenced observation guides: An aid to defining the curriculum and organizing the teaching in communication training programmes. Medical Education, 30(2), 83–89.

    Article  Google Scholar 

  • Levinson, W. (1994). Physician-patient communication. A key to malpractice prevention. Journal of American Medical Association, 272(20), 1619–1620.

    Article  Google Scholar 

  • Lockyer, J., & Lockyer, J. (2003). Multisource feedback in the assessment of physician competencies. Journal of Continuing Education in the Health Professions, 23(1), 4–12.

    Article  Google Scholar 

  • Makoul, G. (2001). Essential elements of communication in medical encounters: The Kalamazoo consensus statement. Academic Medicine, 76(4), 390–393.

    Article  Google Scholar 

  • Massagli, T. L., Carline, J. D., Massagli, T. L., & Carline, J. D. (2007). Reliability of a 360-degree evaluation to assess resident competence. American Journal of Physical Medicine and Rehabilitation, 86(10), 845–852.

    Article  Google Scholar 

  • Mazor, K. M., Ockene, J. K., Rogers, H. J., Carlin, M. M., & Quirk, M. E. (2005). The relationship between checklist scores on a communication OSCE and analogue patients’ perceptions of communication. Advances in Health Sciences Education, 10(1), 37–51.

    Article  Google Scholar 

  • McLeod, P. J., Tamblyn, R., Benaroya, S., & Snell, L. (1994). Faculty ratings of resident humanism predict patient satisfaction ratings in ambulatory medical clinics. Journal of General Internal Medicine, 9(6), 321–326.

    Article  Google Scholar 

  • Miller, G. E. (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65(9 Suppl), S63–S67.

    Article  Google Scholar 

  • Newble, D. (2004). Techniques for measuring clinical competence: Objective structured clinical examinations. Medical Education, 38(2), 199–203.

    Article  Google Scholar 

  • Ramsey, P. G., & Wenrich, M. D. (1999). Peer ratings. An assessment tool whose time has come. Journal of General Internal Medicine, 14(9), 581–582.

    Article  Google Scholar 

  • Ramsey, P. G., Wenrich, M. D., Carline, J. D., Inui, T. S., Larson, E. B., & LoGerfo, J. P. (1993). Use of peer ratings to evaluate physician performance. Journal of American Medical Association, 269(13), 1655–1660.

    Article  Google Scholar 

  • Razavi, D., Merckaert, I., Marchal, S., Libert, Y., Conradt, S., Boniver, J., et al. (2003). How to optimize physicians’ communication skills in cancer care: Results of a randomized study assessing the usefulness of posttraining consolidation workshops. Journal of Clinical Oncology, 21(16), 3141–3149.

    Article  Google Scholar 

  • Regehr, G., MacRae, H., Reznick, R. K., & Szalay, D. (1998). Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Academic Medicine, 73(9), 993–997.

    Article  Google Scholar 

  • Rowland-Morin, P. A., & Carroll, J. G. (1990). Verbal communication skills and patient satisfaction. A study of doctor-patient interviews. Evaluation & the Health Professions, 13(2), 168–185.

    Article  Google Scholar 

  • Schnabl, G. K., Hassard, T. H., & Kopelow, M. L. (1991). The assessment of interpersonal skills using standardized patients. Academic Medicine, 66(9 Suppl), S34–S36.

    Article  Google Scholar 

  • Street, R. L., Jr. (1992). Analyzing communication in medical consultations. Do behavioral measures correspond to patients’ perceptions? Medical Care, 30(11), 976–988.

    Article  Google Scholar 

  • Stump, T. E., Dexter, P. R., Tierney, W. M., & Wolinsky, F. D. (1995). Measuring patient satisfaction with physicians among older and diseased adults in a primary care municipal outpatient setting. An examination of three instruments. Medical Care, 33(9), 958–972.

    Article  Google Scholar 

  • Thomas, P. A., Gebo, K. A., & Hellmann, D. B. (1999). A pilot study of peer review in residency training. Journal of General Internal Medicine, 14(9), 551–554.

    Article  Google Scholar 

  • Turner, J. T., & Dankoski, M. E. (2008). Objective structured clinical exams: A critical review. Family Medicine, 40(8), 574–578.

    Google Scholar 

  • Violato, C., Marini, A., Toews, J., Lockyer, J., & Fidler, H. (1997). Feasibility and psychometric properties of using peers, consulting physicians, co-workers, and patients to assess physicians. Academic Medicine, 72(10 Suppl 1), S82–S84.

    Article  Google Scholar 

Download references

Acknowledgments

We would like to thank all patients, staffs, nurses, and residents who participated in this research.

Author information

Authors and Affiliations

  1. Department of Family Medicine, National Taiwan University College of Medicine and National Taiwan University Hospital, National No. 7, Chung-Shan South Road, Taipei, 10016, Taiwan, ROC

    Kai-Kuen Leung & Wei-Dan Wang

  2. Department of Social Medicine, National Taiwan University College of Medicine and National Taiwan University Hospital, Taipei, Taiwan, ROC

    Wei-Dan Wang & Yen-Yuan Chen

  3. Department of Medical Education, National Taiwan University College of Medicine and National Taiwan University Hospital, Taipei, Taiwan, ROC

    Yen-Yuan Chen

Authors
  1. Kai-Kuen Leung
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei-Dan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yen-Yuan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kai-Kuen Leung.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Leung, KK., Wang, WD. & Chen, YY. Multi-source evaluation of interpersonal and communication skills of family medicine residents. Adv in Health Sci Educ 17, 717–726 (2012). https://doi.org/10.1007/s10459-011-9345-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-011-9345-9

Keywords

Search

Navigation