Advances in Health Sciences Education

, Volume 19, Issue 1, pp 99–113 | Cite as

The Professionalism Assessment of Clinical Teachers (PACT): the reliability and validity of a novel tool to evaluate professional and clinical teaching behaviors

  • Meredith E. Young
  • Sylvia R. Cruess
  • Richard L. Cruess
  • Yvonne Steinert
Article

Abstract

Physicians function as clinicians, teachers, and role models within the clinical environment. Negative learning environments have been shown to be due to many factors, including the presence of unprofessional behaviors among clinical teachers. Reliable and valid assessments of clinical teacher performance, including professional behaviors, may provide a foundation for evidence-based feedback to clinical teachers, enable targeted remediation or recognition, and help to improve the learning environment. However, few tools exist for the evaluation of clinical teachers that include a focus on both professional and clinical teaching behaviors. The Professionalism Assessment of Clinical Teachers (PACT) was developed and implemented at one Canadian institution and was assessed for evidence of reliability and validity. Following each clerkship rotation, students in the 2009–2010 third-year undergraduate clerkship cohort (n = 178) anonymously evaluated a minimum of two clinical teachers using the PACT. 4,715 forms on 567 faculty members were completed. Reliability, validity, and free text comments (present in 45 % of the forms) were examined. An average of 8.6 PACT forms were completed per faculty (range 1–60), with a reliability of 0.31 for 2.9 forms (harmonic mean); 12 forms were necessary for a reliability of 0.65. Global evaluations of teachers aligned with ratings of free-text comments (r = 0.77, p < 0.001). Comment length related negatively with overall rating (r = −0.19, p < 0.001). Mean performance related negatively with variability of performance (r = −0.72, p < 0.001), although this may be related to a ceiling effect. Most faculty members were rated highly; however ‘provided constructive feedback’ was the least well-rated item. Respectful interactions with students appeared to be the most influential item in the global rating of faculty performance. The PACT is a moderately reliable tool for the assessment of professional behaviors of clinical teachers, with evidence supporting its validity.

Keywords

Assessment Professionalism Clinical teaching Faculty development Hidden curriculum Reliability Validity 

References

  1. Blackall, G. F., Melnick, S. A., Shoop, G. H., George, J., Lerner, S. M., Wilson, P. K., et al. (2007). Professionalism in medical education: The development and validation of a survey instrument to assess attitudes toward professionalism. Medical Teacher, 29, E58–E62.CrossRefGoogle Scholar
  2. Boudreau, J. D., Cassell, E. J., & Fuks, A. (2007). A healing curriculum. Medical Education, 41, 1193–1201.Google Scholar
  3. Brainard, A. H., & Bilsen, H. C. (2007). Learning professionalism: A view from the trenches. Academic Medicine, 82, 1010–1014.CrossRefGoogle Scholar
  4. Brennan, R. L. (2001). Manual for urGENOVA (version 2.1). Iowa City, IA: The University of Iowa.Google Scholar
  5. Campbell, J., Richards, S. H., Dickens, A., Greco, M., Narayanan, A., & Brearley, S. (2008). Assessing the professional performance of UK doctors: An evaluation of the utility of the General Medical Council patient and colleague questionnaires. Quality and Safety in Health Care, 17, 187–193.CrossRefGoogle Scholar
  6. Cohen, J. J. (2006). Professionalism in medical education, an American perspective: From evidence to accountability. Medical Education, 40, 607–617.CrossRefGoogle Scholar
  7. Cruess, R. L., & Cruess, S. R. (2009). Principles for designing a program for the teaching and learning of professionalism at the undergraduate level. In R. L. Cruess, S. R. Cruess, & Y. Steinert (Eds.), Teaching medical professionalism (pp. 73–93). Cambridge: Cambridge University Press.Google Scholar
  8. Cruess, R. L., Herold-McIlroy, J., Cruess, S. R., Ginsberg, S., & Steinert, Y. (2006). The P-MEX (professionalism mini evaluation exercise): A preliminary investigation. Academic Medicine, 81, S74–S79.CrossRefGoogle Scholar
  9. Evans, S. M., Berry, J. G., Smith, B. J., Esterman, A., Selim, P., O’Shaughnessy, J., et al. (2006). Attitudes and barriers to incident reporting: A collaborative hospital study. Quality & Safety in Health Care, 15(1), 39–43.CrossRefGoogle Scholar
  10. Feudtner, C., Christakis, D. A., & Christakis, N. A. (1994). Do clinical clerks suffer ethical erosion? Student perceptions of their ethical environment and personal development. Academic Medicine, 69, 670–679.CrossRefGoogle Scholar
  11. Fluit, C. R. M., Bolhuis, S., Grol, R., Loan, R., & Wensing, M. (2010). Assessing the quality of clinical teachers: A systematic review of content and quality of questionnaires for assessing clinical teachers. Journal of General Internal Medicine, 25, 1337–1345.CrossRefGoogle Scholar
  12. Garra, G., Wackett, A., & Thode, H. (2011). Feasibility and reliability of a multisource feedback tool for emergency medicine residents. Journal of Graduate Medical Education, 3, 356–360.CrossRefGoogle Scholar
  13. Ginsburg, S., Regehr, G., Hatala, R., McNaughton, N., Frohna, A., Hodges, B., et al. (2000). Context, conflict, and resolutions: A new conceptual framework for evaluating professionalism. Academic Medicine, 75, 96–511.CrossRefGoogle Scholar
  14. Ginsburg, S., Regehr, G., & Lingard, L. (2004). Basing the evaluation of professionalism on observable behaviors: A precautionary tale. Academic Medicine, 79, S1–S4.CrossRefGoogle Scholar
  15. Hickson, G. B., Pichert, J. W., Webb, L. E., & Gabbe, S. G. (2007). A complementary approach to promoting professionalism: Identifying, measuring, and addressing unprofessional behaviors. Academic Medicine, 82, 1040–1048.CrossRefGoogle Scholar
  16. Hodges, B. D., Ginsburg, S., Cruess, R., Cruess, S., Delport, R., Hafferty, F., et al. (2011). Assessment of professionalism: Recommendations from the Ottawa 2010 conference. Medical Teacher, 33, 354–363.CrossRefGoogle Scholar
  17. Lockyer, J. (2003). Multisource feedback in the assessment of physician competencies. Journal of Continued Education in Health Professions, 23(1), 4–12.CrossRefGoogle Scholar
  18. Lockyer, J. M., Violato, C., & Fidler, H. (2006). A multisource feedback program for anesthesiologists. Canadian Journal of Anaesthesia, 53(1), 33–39.CrossRefGoogle Scholar
  19. Petrusa, E. R. (2002). Clinical performance assessments. In G. R. Norman, C. P. M. van der Vleuten, & D. I. Newble (Eds.), International handbook of research in medical education (pp. 673–709). Dordrecht: Kluwer.CrossRefGoogle Scholar
  20. Prislin, M. D., Lie, D., Shapiro, J., Boker, J., & Radecki, S. (2001). Using standardized patients to assess medical students’ professionalism. Academic Medicine, 76(10 suppl), 90–92.CrossRefGoogle Scholar
  21. Quaintance, J. L., Arnold, L., & Thompson, G. S. (2008). Development of an instrument to measure the climate of professionalism in a clinical teaching environment. Academic Medicine, 83, S5–S8.CrossRefGoogle Scholar
  22. Razack, S., Faremo, S., Drolet, F., Snell, L., Wiseman, J., & Pickering, J. (2009). Multiple mini-interviews versus traditional interviews: Stakeholder acceptability comparison. Medical Education, 43, 993–1000.CrossRefGoogle Scholar
  23. Shannon, H. S., & Norman, G. R. (2009). Deriving the factor structure of safety climate scales. Safety Science, 27, 327–329.CrossRefGoogle Scholar
  24. Snell, L. (2009). Teaching professionalism and fostering professional values during residency: The McGill experience. In R. L. Cruess, S. R. Cruess, & Y. Steinert (Eds.), Teaching medical professionalism (pp. 246–263). Cambridge: Cambridge University Press.Google Scholar
  25. Srinivasan, M., Li, S. T., Meyers, F. J., Pratt, D., Collins, J. B., Braddock, C., et al. (2011). “Teaching as a competency”: Competencies for medical educators. Academic Medicine, 86, 1211–1220.CrossRefGoogle Scholar
  26. Stern, D. T. (Ed.). (2005). Measuring medical professionalism. New York: Oxford University Press.Google Scholar
  27. Streiner, D., & Norman, G. (2008). Health measurement scales: A practical guide to their development and use. USA: Oxford University Press.CrossRefGoogle Scholar
  28. Sutkin, G., Littlefield, J. H., & Laube, D. W. (2009). Nursing staff assessment of residents’ professionalism and communication skills. Medical Education, 43(11), 1104.CrossRefGoogle Scholar
  29. Sutkin, G., Wagner, E., Harris, I., & Schiffer, R. (2008). What makes a good clinical teacher in medicine? A review of the literature. Academic Medicine, 83, 452–466.CrossRefGoogle Scholar
  30. Todhunter, S., Cruess, S. R., Cruess, R. L., Young, M., & Steinert, Y. (2011). Developing and piloting a form for student assessment of faculty professionalism. Advances in Health Sciences Education, 16, 223–238.CrossRefGoogle Scholar
  31. Tromp, F., Vernooij-Dassen, M., Kramer, A., Grol, R., & Bottema, K. (2010). Behavioral elements of professionalism: Assessment of a fundamental concept in medical care. Medical Teacher, 32, E161–E169.CrossRefGoogle Scholar
  32. Wilkinson, T. J., Wade, W. B., & Knock, L. D. (2009). A blueprint to assess professionalism: Results of a systematic review. Academic Medicine, 84(5), 551–558.CrossRefGoogle Scholar
  33. Wood, T. (2009). Assessment not only drives learning, it may also help learning. Medical Education, 43, 5–6.CrossRefGoogle Scholar
  34. Wood, J., Collins, J., Burnside, E. S., Albanese, M. A., Propeck, P. A., Kelcz, F., et al. (2004). Patient, faculty and self-assessment of radiology resident performance: A 360-degree method of measuring professionalism and interpersonal/communication skills. Academic Radiology, 11, 931–939.Google Scholar
  35. Wright, S. M., Kern, D. E., Kolodner, K., Howard, D. M., & Brancati, F. L. (1998). Attributes of excellent attending-physician role models. New England Journal of Medicine, 339, 1986–1993.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Meredith E. Young
    • 1
  • Sylvia R. Cruess
    • 1
  • Richard L. Cruess
    • 2
  • Yvonne Steinert
    • 3
  1. 1.Department of Medicine, Centre for Medical Education, Faculty of MedicineMcGill UniversityMontrealCanada
  2. 2.Department of Surgery, Centre for Medical EducationMcGill UniversityMontrealCanada
  3. 3.Department of Family Medicine, Centre for Medical EducationMcGill UniversityMontrealCanada

Personalised recommendations