Abstract
The purpose of this study was to provide validity and feasibility evidence in measuring professionalism using the Professionalism Mini-Evaluation Exercise (P-MEX) scores as part of a residency admissions process. In 2012 and 2013, three standardized-patient-based P-MEX encounters were administered to applicants invited for an interview at the University of Geneva Pediatrics Residency Program. Validity evidence was gathered for P-MEX content (item analysis); response process (qualitative feedback); internal structure (inter-rater reliability with intraclass correlation and Generalizability); relations to other variables (correlations); and consequences (logistic regression to predict admission). To improve reliability, Kane’s formula was used to create an applicant composite score using P-MEX, structured letter of recommendation (SLR), and structured interview (SI) scores. Applicant rank lists using composite scores versus faculty global ratings were compared using the Wilcoxon signed-rank test. Seventy applicants were assessed. Moderate associations were found between pairwise correlations of P-MEX scores and SLR (r = 0.25, P = .036), SI (r = 0.34, P = .004), and global ratings (r = 0.48, P < .001). Generalizability of the P-MEX using three cases was moderate (G-coefficient = 0.45). P-MEX scores had the greatest correlation with acceptance (r = 0.56, P < .001), were the strongest predictor of acceptance (OR 4.37, P < .001), and increased pseudo R-squared by 0.20 points. Including P-MEX scores increased composite score reliability from 0.51 to 0.74. Rank lists of applicants using composite score versus global rating differed significantly (z = 5.41, P < .001). Validity evidence supports the use of P-MEX scores to improve the reliability of the residency admissions process by improving applicant composite score reliability.
Similar content being viewed by others
References
Adams, K. E., Emmons, S., & Romm, J. (2008). How resident unprofessional behavior is identified and managed: A program director survey. American Journal of Obstetrics and Gynecology, 198(6), 692.e1–692.e5. doi:10.1016/j.ajog.2008.03.023. discussion 692 e694-695.
American educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association
Arnold, L. (2002). Assessing professional behavior: Yesterday, today, and tomorrow. Academic medicine: Journal of the Association of American Medical Colleges, 77(6), 502–515.
Arnold, L., & Stern, D. T. (2006). What is medical professionalism. Measuring Medical Professionalism, 10016, 15–38.
Birden, H., Glass, N., Wilson, I., Harrison, M., Usherwood, T., & Nass, D. (2013). Teaching professionalism in medical education: A Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 25. Medical Teacher, 35(7), e1252–e1266. doi:10.3109/0142159x.2013.789132.
Brennan, R. L. (2001). Generalizability Theory. Berlin: Springer.
Brennan, R. L., Gao, X., & Colton, D. A. (1995). Generalizability analyses of work keys listening and writing tests. Educational and Psychological Measurement, 55(2), 157–176. doi:10.1177/0013164495055002001.
Brenner, A. M., Mathai, S., Jain, S., & Mohl, P. C. (2010). Can we predict “problem residents?”. In Meeting of the American Association of Directors of Psychiatry Residency Training, 2007, San Juan, Puerto Rico; The results of this study were presented in a workshop at the aforementioned conference 85(7), 1147–1151.
Corcoran, J., Downing, S. M., Tekian, A., & DaRosa, D. A. (2009). Composite score validity in clerkship grading. Academic medicine: Journal of the Association of American Medical Colleges, 84(10 Suppl), S120–S123. doi:10.1097/ACM.0b013e3181b37009.
Cruess, R., McIlroy, J. H., Cruess, S., Ginsburg, S., & Steinert, Y. (2006). The Professionalism Mini-evaluation Exercise: A preliminary investigation. Academic medicine: Journal of the Association of American Medical Colleges, 81(10 Suppl), S74–S78.
Demaurex, F. V. N. (2013). Patients simulés/standardisés. In G. S. A. J. Granry (Ed.), La simulation en santé: De la théorie à la pratique. Paris: Springer.
Dore, K. L., Kreuger, S., Ladhani, M., Rolfson, D., Kurtz, D., Kulasegaram, K., & Reiter, H. I. (2010). The reliability and acceptability of the multiple mini-interview as a selection instrument for postgraduate admissions. Academic medicine: Journal of the Association of American Medical Colleges, 85(10 Suppl), S60–S63. doi:10.1097/ACM.0b013e3181ed442b.
Downing, S. M. (2003). Validity: On meaningful interpretation of assessment data. Medical Education, 37(9), 830–837.
Durning, S. J., Pangaro, L. N., Lawrence, L. L., Waechter, D., McManigle, J., & Jackson, J. L. (2005). The feasibility, reliability, and validity of a program director’s (supervisor’s) evaluation form for medical school graduates. Academic medicine: Journal of the Association of American Medical Colleges, 80(10), 964–968.
Eva, K. W., Reiter, H. I., Trinh, K., Wasi, P., Rosenfeld, J., & Norman, G. R. (2009). Predictive validity of the multiple mini-interview for selecting medical trainees. Medical Education, 43(8), 767–775. doi:10.1111/j.1365-2923.2009.03407.x.
Eva, K. W., Rosenfeld, J., Reiter, H. I., & Norman, G. R. (2004). An admissions OSCE: The multiple mini-interview. Medical Education, 38(3), 314–326.
Ginsburg, S., Bernabeo, E., Ross, K. M., & Holmboe, E. S. (2012). “It depends”: Results of a qualitative study investigating how practicing internists approach professional dilemmas. Academic medicine: Journal of the Association of American Medical Colleges, 87(12), 1685–1693. doi:10.1097/ACM.0b013e3182736dfc.
Ginsburg, S., Regehr, G., Hatala, R., McNaughton, N., Frohna, A., Hodges, B., & Stern, D. (2000). Context, conflict, and resolution: A new conceptual framework for evaluating professionalism. Academic medicine: Journal of the Association of American Medical Colleges, 75(10 Suppl), S6–S11.
Girzadas, D. V, Jr, Harwood, R. C., Dearie, J., & Garrett, S. (1998). A comparison of standardized and narrative letters of recommendation. Academic Emergency Medicine, 5(11), 1101–1104.
Goldie, J. (2012). Assessment of professionalism: A consolidation of current thinking. Medical Teacher,. doi:10.3109/0142159X.2012.714888.
Greenburg, D. L., Durning, S. J., Cohen, D. L., Cruess, D., & Jackson, J. L. (2007). Identifying medical students likely to exhibit poor professionalism and knowledge during internship. Journal of General Internal Medicine, 22(12), 1711–1717. doi:10.1007/s11606-007-0405-z.
Hodges, B. D., Ginsburg, S., Cruess, R., Cruess, S., Delport, R., Hafferty, F., & Wade, W. (2011). Assessment of professionalism: Recommendations from the Ottawa 2010 conference. Medical Teacher, 33(5), 354–363. doi:10.3109/0142159x.2011.577300.
Kane, M., & Case, S. M. (2004). The reliability and validity of weighted composite scores. Applied Measurement in Education, 17(3), 221–240.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174.
Lievens, F., Buyse, T., & Sackett, P. R. (2005). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90(3), 442–452. doi:10.1037/0021-9010.90.3.442.
Messick, S. (1995). Standards of validity and the validity of standards in performance asessment. Educational Measurement: Issues and Practice, 14(4), 5–8.
Metro, D. G., Talarico, J. F., Patel, R. M., & Wetmore, A. L. (2005). The resident application process and its correlation to future performance as a resident. Anesthesia and Analgesia, 100(2), 502–505. doi:10.1213/01.ANE.0000154583.47236.5F.
Nasca T. (2010). Graduate Medical Education in the United States: Vision and General Directions for the Next 10 Years. Washington, DC: Association of American Medical Colleges, 7 Nov 2010.
Norcini, J. J., Blank, L. L., Arnold, G. K., & Kimball, H. R. (1995). The mini-CEX (clinical evaluation exercise): A preliminary investigation. Annals of Internal Medicine, 123(10), 795–799.
Olawaiye, A., Yeh, J., & Withiam-Leitch, M. (2006). Resident selection process and prediction of clinical performance in an obstetrics and gynecology program. Teaching and Learning in Medicine, 18(4), 310–315. doi:10.1207/s15328015tlm1804_6.
O’Sullivan, H., van Mook, W., Fewtrell, R., & Wass, V. (2012). Integrating professionalism into the curriculum: AMEE Guide No. 61. Medical Teacher, 34(2), e64–e77. doi:10.3109/0142159x.2012.655610.
Papp, K., Polk, H., & Richardson, J. (1997). The relationship between criteria used to select residents and performance during residency. American Journal of Surgery, 173, 326–329.
Patterson, F., Ashworth, V., Zibarras, L., Coan, P., Kerrin, M., & O’Neill, P. (2012). Evaluations of situational judgement tests to assess non-academic attributes in selection. Medical Education, 46(9), 850–868. doi:10.1111/j.1365-2923.2012.04336.x.
Randall, R., Davies, H., Patterson, F., & Farrell, K. (2006). Selecting doctors for postgraduate training in paediatrics using a competency based assessment centre. Archives of Disease in Childhood, 91(5), 444–448. doi:10.1136/adc.2005.076653.
Rodriguez, E., Siegelman, J., Leone, K., & Kessler, C. (2012). Assessing professionalism: Summary of the working group on assessment of observable learner performance. Academic Emergency Medicine, 19(12), 1372–1378. doi:10.1111/acem.12031.
Stern, D. T. (2006). Measuring medical professionalism. Oxford: Oxford University Press.
Strand, E. A., Moore, E., & Laube, D. W. (2011). Can a structured, behavior-based interview predict future resident success? American Journal of Obstetrics and Gynecology, 204(5), 446.e1–446.e13.
Swanson, W. S., Harris, M. C., Master, C., Gallagher, P. R., Mauro, A. E., & Ludwig, S. (2005). The impact of the interview in pediatric residency selection. Ambulatory pediatrics: The official journal of the Ambulatory Pediatric Association, 5(4), 216–220. doi:10.1367/A04-149R1.1.
Swanwick, T., & Association for the Study of Medical Education. (2010). Understanding medical education: Evidence, theory and practice (1st ed.). Chichester, West Sussex: Wiley.
Swiss Catalogue of Learning Objectives for Undergraduate Medical Training. (2008). Working Group under a Mandate of the Joint Commission of the Swiss Medical Schools.
Tsugawa, Y., Ohbu, S., Cruess, R., Cruess, S., Okubo, T., Takahashi, O., & Fukui, T. (2011). Introducing the professionalism mini-evaluation exercise (P-MEX) in Japan: Results from a multicenter, cross-sectional study. Academic medicine: Journal of the Association of American Medical Colleges, 86(8), 1026–1031. doi:10.1097/ACM.0b013e3182222ba0.
Wilkinson, T. J., Wade, W. B., & Knock, L. D. (2009). A blueprint to assess professionalism: Results of a systematic review. Academic medicine: Journal of the Association of American Medical Colleges, 84(5), 551–558. doi:10.1097/ACM.0b013e31819fbaa2.
Yao, D. C., & Wright, S. M. (2000). National survey of internal medicine residency program directors regarding problem residents. JAMA, the Journal of the American Medical Association, 284(9), 1099–1104.
Zbieranowski, I., Takahashi, S. G., Verma, S., & Spadafora, S. M. (2013). Remediation of residents in difficulty: A retrospective 10-year review of the experience of a postgraduate board of examiners. Academic medicine: Journal of the Association of American Medical Colleges, 88(1), 111–116. doi:10.1097/ACM.0b013e3182764cb6.
Acknowledgments
The authors wish to thank Dr. Florence Demaurex for her assistance in training the SPs.
Funding
This study was funded in part by a Grant from the Global Pediatric Education Consortium (GPEC).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
None.
Ethical approval
The Institutional Review Board at the University Hospital of Geneva and the University of Illinois at Chicago approved this study.
Rights and permissions
About this article
Cite this article
Bajwa, N.M., Yudkowsky, R., Belli, D. et al. Improving the residency admissions process by integrating a professionalism assessment: a validity and feasibility study. Adv in Health Sci Educ 22, 69–89 (2017). https://doi.org/10.1007/s10459-016-9683-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-016-9683-8