Skip to main content
SpringerLink
Log in

Improving the residency admissions process by integrating a professionalism assessment: a validity and feasibility study

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

The purpose of this study was to provide validity and feasibility evidence in measuring professionalism using the Professionalism Mini-Evaluation Exercise (P-MEX) scores as part of a residency admissions process. In 2012 and 2013, three standardized-patient-based P-MEX encounters were administered to applicants invited for an interview at the University of Geneva Pediatrics Residency Program. Validity evidence was gathered for P-MEX content (item analysis); response process (qualitative feedback); internal structure (inter-rater reliability with intraclass correlation and Generalizability); relations to other variables (correlations); and consequences (logistic regression to predict admission). To improve reliability, Kane’s formula was used to create an applicant composite score using P-MEX, structured letter of recommendation (SLR), and structured interview (SI) scores. Applicant rank lists using composite scores versus faculty global ratings were compared using the Wilcoxon signed-rank test. Seventy applicants were assessed. Moderate associations were found between pairwise correlations of P-MEX scores and SLR (r = 0.25, P = .036), SI (r = 0.34, P = .004), and global ratings (r = 0.48, P < .001). Generalizability of the P-MEX using three cases was moderate (G-coefficient = 0.45). P-MEX scores had the greatest correlation with acceptance (r = 0.56, P < .001), were the strongest predictor of acceptance (OR 4.37, P < .001), and increased pseudo R-squared by 0.20 points. Including P-MEX scores increased composite score reliability from 0.51 to 0.74. Rank lists of applicants using composite score versus global rating differed significantly (z = 5.41, P < .001). Validity evidence supports the use of P-MEX scores to improve the reliability of the residency admissions process by improving applicant composite score reliability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Adams, K. E., Emmons, S., & Romm, J. (2008). How resident unprofessional behavior is identified and managed: A program director survey. American Journal of Obstetrics and Gynecology, 198(6), 692.e1–692.e5. doi:10.1016/j.ajog.2008.03.023. discussion 692 e694-695.

    Article  Google Scholar 

  • American educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association

  • Arnold, L. (2002). Assessing professional behavior: Yesterday, today, and tomorrow. Academic medicine: Journal of the Association of American Medical Colleges, 77(6), 502–515.

    Article  Google Scholar 

  • Arnold, L., & Stern, D. T. (2006). What is medical professionalism. Measuring Medical Professionalism, 10016, 15–38.

    Google Scholar 

  • Birden, H., Glass, N., Wilson, I., Harrison, M., Usherwood, T., & Nass, D. (2013). Teaching professionalism in medical education: A Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 25. Medical Teacher, 35(7), e1252–e1266. doi:10.3109/0142159x.2013.789132.

    Article  Google Scholar 

  • Brennan, R. L. (2001). Generalizability Theory. Berlin: Springer.

    Book  Google Scholar 

  • Brennan, R. L., Gao, X., & Colton, D. A. (1995). Generalizability analyses of work keys listening and writing tests. Educational and Psychological Measurement, 55(2), 157–176. doi:10.1177/0013164495055002001.

    Article  Google Scholar 

  • Brenner, A. M., Mathai, S., Jain, S., & Mohl, P. C. (2010). Can we predict “problem residents?”. In Meeting of the American Association of Directors of Psychiatry Residency Training, 2007, San Juan, Puerto Rico; The results of this study were presented in a workshop at the aforementioned conference 85(7), 1147–1151.

  • Corcoran, J., Downing, S. M., Tekian, A., & DaRosa, D. A. (2009). Composite score validity in clerkship grading. Academic medicine: Journal of the Association of American Medical Colleges, 84(10 Suppl), S120–S123. doi:10.1097/ACM.0b013e3181b37009.

    Article  Google Scholar 

  • Cruess, R., McIlroy, J. H., Cruess, S., Ginsburg, S., & Steinert, Y. (2006). The Professionalism Mini-evaluation Exercise: A preliminary investigation. Academic medicine: Journal of the Association of American Medical Colleges, 81(10 Suppl), S74–S78.

    Article  Google Scholar 

  • Demaurex, F. V. N. (2013). Patients simulés/standardisés. In G. S. A. J. Granry (Ed.), La simulation en santé: De la théorie à la pratique. Paris: Springer.

    Google Scholar 

  • Dore, K. L., Kreuger, S., Ladhani, M., Rolfson, D., Kurtz, D., Kulasegaram, K., & Reiter, H. I. (2010). The reliability and acceptability of the multiple mini-interview as a selection instrument for postgraduate admissions. Academic medicine: Journal of the Association of American Medical Colleges, 85(10 Suppl), S60–S63. doi:10.1097/ACM.0b013e3181ed442b.

    Article  Google Scholar 

  • Downing, S. M. (2003). Validity: On meaningful interpretation of assessment data. Medical Education, 37(9), 830–837.

    Article  Google Scholar 

  • Durning, S. J., Pangaro, L. N., Lawrence, L. L., Waechter, D., McManigle, J., & Jackson, J. L. (2005). The feasibility, reliability, and validity of a program director’s (supervisor’s) evaluation form for medical school graduates. Academic medicine: Journal of the Association of American Medical Colleges, 80(10), 964–968.

    Article  Google Scholar 

  • Eva, K. W., Reiter, H. I., Trinh, K., Wasi, P., Rosenfeld, J., & Norman, G. R. (2009). Predictive validity of the multiple mini-interview for selecting medical trainees. Medical Education, 43(8), 767–775. doi:10.1111/j.1365-2923.2009.03407.x.

    Article  Google Scholar 

  • Eva, K. W., Rosenfeld, J., Reiter, H. I., & Norman, G. R. (2004). An admissions OSCE: The multiple mini-interview. Medical Education, 38(3), 314–326.

    Article  Google Scholar 

  • Ginsburg, S., Bernabeo, E., Ross, K. M., & Holmboe, E. S. (2012). “It depends”: Results of a qualitative study investigating how practicing internists approach professional dilemmas. Academic medicine: Journal of the Association of American Medical Colleges, 87(12), 1685–1693. doi:10.1097/ACM.0b013e3182736dfc.

    Article  Google Scholar 

  • Ginsburg, S., Regehr, G., Hatala, R., McNaughton, N., Frohna, A., Hodges, B., & Stern, D. (2000). Context, conflict, and resolution: A new conceptual framework for evaluating professionalism. Academic medicine: Journal of the Association of American Medical Colleges, 75(10 Suppl), S6–S11.

    Article  Google Scholar 

  • Girzadas, D. V, Jr, Harwood, R. C., Dearie, J., & Garrett, S. (1998). A comparison of standardized and narrative letters of recommendation. Academic Emergency Medicine, 5(11), 1101–1104.

    Article  Google Scholar 

  • Goldie, J. (2012). Assessment of professionalism: A consolidation of current thinking. Medical Teacher,. doi:10.3109/0142159X.2012.714888.

    Google Scholar 

  • Greenburg, D. L., Durning, S. J., Cohen, D. L., Cruess, D., & Jackson, J. L. (2007). Identifying medical students likely to exhibit poor professionalism and knowledge during internship. Journal of General Internal Medicine, 22(12), 1711–1717. doi:10.1007/s11606-007-0405-z.

    Article  Google Scholar 

  • Hodges, B. D., Ginsburg, S., Cruess, R., Cruess, S., Delport, R., Hafferty, F., & Wade, W. (2011). Assessment of professionalism: Recommendations from the Ottawa 2010 conference. Medical Teacher, 33(5), 354–363. doi:10.3109/0142159x.2011.577300.

    Article  Google Scholar 

  • Kane, M., & Case, S. M. (2004). The reliability and validity of weighted composite scores. Applied Measurement in Education, 17(3), 221–240.

    Article  Google Scholar 

  • Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics 33(1), 159–174.

    Article  Google Scholar 

  • Lievens, F., Buyse, T., & Sackett, P. R. (2005). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90(3), 442–452. doi:10.1037/0021-9010.90.3.442.

    Article  Google Scholar 

  • Messick, S. (1995). Standards of validity and the validity of standards in performance asessment. Educational Measurement: Issues and Practice, 14(4), 5–8.

    Article  Google Scholar 

  • Metro, D. G., Talarico, J. F., Patel, R. M., & Wetmore, A. L. (2005). The resident application process and its correlation to future performance as a resident. Anesthesia and Analgesia, 100(2), 502–505. doi:10.1213/01.ANE.0000154583.47236.5F.

    Article  Google Scholar 

  • Nasca T. (2010). Graduate Medical Education in the United States: Vision and General Directions for the Next 10 Years. Washington, DC: Association of American Medical Colleges, 7 Nov 2010.

  • Norcini, J. J., Blank, L. L., Arnold, G. K., & Kimball, H. R. (1995). The mini-CEX (clinical evaluation exercise): A preliminary investigation. Annals of Internal Medicine, 123(10), 795–799.

    Article  Google Scholar 

  • Olawaiye, A., Yeh, J., & Withiam-Leitch, M. (2006). Resident selection process and prediction of clinical performance in an obstetrics and gynecology program. Teaching and Learning in Medicine, 18(4), 310–315. doi:10.1207/s15328015tlm1804_6.

    Article  Google Scholar 

  • O’Sullivan, H., van Mook, W., Fewtrell, R., & Wass, V. (2012). Integrating professionalism into the curriculum: AMEE Guide No. 61. Medical Teacher, 34(2), e64–e77. doi:10.3109/0142159x.2012.655610.

    Article  Google Scholar 

  • Papp, K., Polk, H., & Richardson, J. (1997). The relationship between criteria used to select residents and performance during residency. American Journal of Surgery, 173, 326–329.

    Article  Google Scholar 

  • Patterson, F., Ashworth, V., Zibarras, L., Coan, P., Kerrin, M., & O’Neill, P. (2012). Evaluations of situational judgement tests to assess non-academic attributes in selection. Medical Education, 46(9), 850–868. doi:10.1111/j.1365-2923.2012.04336.x.

    Article  Google Scholar 

  • Randall, R., Davies, H., Patterson, F., & Farrell, K. (2006). Selecting doctors for postgraduate training in paediatrics using a competency based assessment centre. Archives of Disease in Childhood, 91(5), 444–448. doi:10.1136/adc.2005.076653.

    Article  Google Scholar 

  • Rodriguez, E., Siegelman, J., Leone, K., & Kessler, C. (2012). Assessing professionalism: Summary of the working group on assessment of observable learner performance. Academic Emergency Medicine, 19(12), 1372–1378. doi:10.1111/acem.12031.

    Article  Google Scholar 

  • Stern, D. T. (2006). Measuring medical professionalism. Oxford: Oxford University Press.

    Google Scholar 

  • Strand, E. A., Moore, E., & Laube, D. W. (2011). Can a structured, behavior-based interview predict future resident success? American Journal of Obstetrics and Gynecology, 204(5), 446.e1–446.e13.

    Article  Google Scholar 

  • Swanson, W. S., Harris, M. C., Master, C., Gallagher, P. R., Mauro, A. E., & Ludwig, S. (2005). The impact of the interview in pediatric residency selection. Ambulatory pediatrics: The official journal of the Ambulatory Pediatric Association, 5(4), 216–220. doi:10.1367/A04-149R1.1.

    Article  Google Scholar 

  • Swanwick, T., & Association for the Study of Medical Education. (2010). Understanding medical education: Evidence, theory and practice (1st ed.). Chichester, West Sussex: Wiley.

    Google Scholar 

  • Swiss Catalogue of Learning Objectives for Undergraduate Medical Training. (2008). Working Group under a Mandate of the Joint Commission of the Swiss Medical Schools.

  • Tsugawa, Y., Ohbu, S., Cruess, R., Cruess, S., Okubo, T., Takahashi, O., & Fukui, T. (2011). Introducing the professionalism mini-evaluation exercise (P-MEX) in Japan: Results from a multicenter, cross-sectional study. Academic medicine: Journal of the Association of American Medical Colleges, 86(8), 1026–1031. doi:10.1097/ACM.0b013e3182222ba0.

    Article  Google Scholar 

  • Wilkinson, T. J., Wade, W. B., & Knock, L. D. (2009). A blueprint to assess professionalism: Results of a systematic review. Academic medicine: Journal of the Association of American Medical Colleges, 84(5), 551–558. doi:10.1097/ACM.0b013e31819fbaa2.

    Article  Google Scholar 

  • Yao, D. C., & Wright, S. M. (2000). National survey of internal medicine residency program directors regarding problem residents. JAMA, the Journal of the American Medical Association, 284(9), 1099–1104.

    Article  Google Scholar 

  • Zbieranowski, I., Takahashi, S. G., Verma, S., & Spadafora, S. M. (2013). Remediation of residents in difficulty: A retrospective 10-year review of the experience of a postgraduate board of examiners. Academic medicine: Journal of the Association of American Medical Colleges, 88(1), 111–116. doi:10.1097/ACM.0b013e3182764cb6.

    Article  Google Scholar 

Download references

Acknowledgments

The authors wish to thank Dr. Florence Demaurex for her assistance in training the SPs.

Funding

This study was funded in part by a Grant from the Global Pediatric Education Consortium (GPEC).

Author information

Authors and Affiliations

  1. Department of General Pediatrics, Hôpital des Enfants, Hôpitaux Universitaires de Genève, Geneva, Switzerland

    Nadia M. Bajwa & Dominique Belli

  2. Department of Medical Education, College of Medicine, University of Illinois at Chicago, Chicago, IL, USA

    Rachel Yudkowsky & Yoon Soo Park

  3. Development and Research in Medical Education, Faculty of Medicine, University of Geneva, Geneva, Switzerland

    Nu Viet Vu

  4. Département de l’enfant et de l’adolescent, Service de pédiatrie générale, Hôpital des Enfants, Hôpitaux Universitaires de Genève, Rue Willy Donzé 6, 1211, Geneva 14, Switzerland

    Nadia M. Bajwa

Authors
  1. Nadia M. Bajwa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rachel Yudkowsky
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dominique Belli
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nu Viet Vu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yoon Soo Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nadia M. Bajwa.

Ethics declarations

Conflict of interest

None.

Ethical approval

The Institutional Review Board at the University Hospital of Geneva and the University of Illinois at Chicago approved this study.

Appendices

Appendix 1

See Table 6.

Table 6 Blueprint for the residency admissions process
Full size table

Appendix 2: Sample standardized letter of recommendation

figure afigure a

Appendix 3

See Table 7.

Table 7 Example of structured interview question and possible responses
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bajwa, N.M., Yudkowsky, R., Belli, D. et al. Improving the residency admissions process by integrating a professionalism assessment: a validity and feasibility study. Adv in Health Sci Educ 22, 69–89 (2017). https://doi.org/10.1007/s10459-016-9683-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-016-9683-8

Keywords

Search

Navigation