Skip to main content
SpringerLink
Log in

Modeling relationships between traditional preadmission measures and clinical skills performance on a medical licensure examination

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Medical schools employ a variety of preadmission measures to select students most likely to succeed in the program. The Medical College Admission Test (MCAT) and the undergraduate college grade point average (uGPA) are two academic measures typically used to select students in medical school. The assumption that presently used preadmission measures can predict clinical skill performance on a medical licensure examination was evaluated within a validity argument framework (Kane 1992). A hierarchical generalized linear model tested relationships between the log-odds of failing a high-stakes medical licensure performance examination and matriculant academic and non-academic preadmission measures, controlling for student-and school-variables. Data includes 3,189 matriculants from 22 osteopathic medical schools tested in 2009–2010. Unconditional unit-specific model expected average log-odds of failing the examination across medical schools is −3.05 (se = 0.11) or 5%. Student-level estimated coefficients for MCAT Verbal Reasoning scores (0.03), Physical Sciences scores (0.05), Biological Sciences scores (0.04), uGPAscience (0.07), and uGPAnon-science (0.26) lacked association with the log-odds of failing the COMLEX-USA Level 2-PE, controlling for all other predictors in the model. Evidence from this study shows that present preadmission measures of academic ability are not related to later clinical skill performance. Given that clinical skill performance is an important part of medical practice, selection measures should be developed to identify students who will be successful in communication and be able to demonstrate the ability to systematically collect a medical history, perform a physical examination, and synthesize this information to diagnose and manage patient conditions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Accreditation Council for Graduate Medical Education. (2005). The ACGME outcome project: An introduction (pp. 1–39). Retrieved from https://www.acgme.org/outcome/project/OPintrorev1_7-05.ppt.

  • Agresti, A. (1996). An introduction to categorical data analysis. New York: Wiley.

    Google Scholar 

  • Albanese, M. A., Snow, M. H., Skochelak, S. E., Huggett, K. N., & Farrell, P. M. (2003). Assessing personal qualities in medical school admissions. Academic Medicine, 78(3), 313–321.

    Article  Google Scholar 

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

    Google Scholar 

  • Baker, H. H., Cope, M. K., Adelman, M. D., Schuler, S., Foster, R. W., & Gimpel, J. R. (2006). Relationships between scores on the COMLEX-USA Level 2-Peformance Evaluation and selected school-based performance measures. The Journal of the American Osteopathic Association, 106(5), 290–295.

    Google Scholar 

  • Boulet, J. R., Smee, S. M., Dillon, G. F., & Gimpel, J. R. (2009). The use of standardized patient assessments for certification and licensure decisions. Simulation in Health Care, 4(1), 35–42.

    Article  Google Scholar 

  • Commission on Osteopathic College Accreditation. (2011). Accreditation of colleges of osteopathic medicine: COM accreditation standards and proceedures (pp. 1–85). Chicago, IL: American Osteopathic Association.

    Google Scholar 

  • Cronbach, L. J. (1989). Construct validation after thirty years. In R. L. Linn (Ed.), Intelligence: Measurement, theory, and public policy (pp. 147–171). Urbana, IL: Univerisity of Illinois Press.

    Google Scholar 

  • Cuddy, M. M., Swanson, D. B., & Clauser, B. E. (2007). A multilevel analysis of the relationsips between examinee gender and United States medical licensing exam (USMLE) Step 2 CK content area perfomrance. Academic Medicine, 82(10), 89–93.

    Article  Google Scholar 

  • Donnon, T., Oddone, E., & Violato, C. (2007). The predictive validity of the MCAT for medical school performance and medical board licensing examinations: A meta-analysis of published research. Academic Medicine, 82(1), 100–106.

    Article  Google Scholar 

  • Eva, K. W., & Reiter, H. I. (2004). Where judgement fails: pitfalls in the selection process for medical personnel. Advances in Health Sciences Education, 9(2), 161–174. doi:10.1023/B:AHSE.0000027479.14637.6c.

    Article  Google Scholar 

  • Eva, K. W., Reiter, H. I., Rosenfeld, J., & Norman, G. (2004a). The relationship between interviewer characteristics and ratings assigned during a multiple mini-interview. Academic Medicine, 79, 602–608.

    Article  Google Scholar 

  • Eva, K. W., Reiter, H. I., Trinh, K., Wasi, P., Rosenfeld, J., & Norman, G. R. (2009). Predictive validity of the multiple mini-interview for selecting medical trainees. Medical Education, 43, 767–775. doi:10.1111/j.1365-2923.2009.03407.x.

    Article  Google Scholar 

  • Eva, K. W., Rosenfeld, J., Reiter, H. I., & Norman, G. R. (2004b). An admissions OSCE: The multiple mini-interview. Medical Education, 38(3), 314–326. doi:10.1046/j.1365-2923.2004.01776.x.

    Article  Google Scholar 

  • Gimpel, J. R., Boulet, J. R., & Weidner, A. C. (2006). Survey on the clinical skills of osteopathic medical students. The Journal of the American Osteopathic Association, 106, 296–301.

    Google Scholar 

  • Hamdy, H., Prasad, K., Brownell, A. M., Scherpbier, A., Williams, R., Zwierstra, R., et al. (2006). BEME systematic review: Predictive values of measurements obtained in medical schools and future performance in medical practice. Medical Teacher, 28(2), 103–116. doi:10.1080/01421590600622723.

    Article  Google Scholar 

  • Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535.

    Article  Google Scholar 

  • Kane, M. (2004). Certification testing as an illustration of argument-based validation. Measurement Interdisciplinary Research and Perspectives, 2(3), 135–170.

    Article  Google Scholar 

  • Kane, M., Crooks, T., & Cohen, A. (1999). Validating measures of performance. Educational Measurement Issues and Practice, 18(2), 5–17. doi:10.1111/j.1745-3992.1999.tb00010.x.

    Article  Google Scholar 

  • Kreiter, C. D., & Kreiter, Y. (2007). A validity generalization perspective on the ability of undergraduate GPA and the medical college admission test to predict important outcomes. Teaching and Learning in Medicine, 19(2), 95–100.

    Article  Google Scholar 

  • Kulatunga-Moruzi, C., & Norman, G. R. (2002). Validity of admissions measures in predicting performance outcomes: The contribution of cognitive and no-cognitive dimensions. Teaching and Learning in Medicine, 14(1), 34–42.

    Article  Google Scholar 

  • Langenau, E. E., Dyer, C., Roberts, W. L., Wilson, C., & Gimpel, J. (2010). Five-year summary of COMLEX-USA Level 2-PE examinee performance and survey data. The Journal of the American Osteopathic Association, 110(3), 114–125.

    Google Scholar 

  • Long, J. S. (1997). Regression models for categorical and limited dependent variables. Thousand Oaks, CA: Sage.

    Google Scholar 

  • Margolis, M. J., Clauser, B. E., Swanson, D. B., & Boulet, J. R. (2003). Analysis of the relationship between score components on a standardized patient clinical skills examination. Academic Medicine, 78(10), S68–S71. [Supplement issue].

    Article  Google Scholar 

  • Mitchell, K. J. (1990). Traditional predictors of performance in medical school. Academic Medicine, 65(3), 149–158.

    Article  Google Scholar 

  • National Board of Osteopathic Medical Examiners. (2010). 2010–2011 Orientation guide COMLEX-USA Level 2-PE (pp. 1–14). Retrieved from http://www.nbome.org/docs/PEOrientationGuide.pdf.

  • National Board of Osteopathic Medical Examiners. (2011). COMLEX computer based testing (CBT). Retrieved from http://www.nbome.org/comlex-cbt.asp?m=can.

  • Raudenbush, S. W., & Bryke, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.

    Google Scholar 

  • Roberts, J., & Norman, G. (1990). Reliability and learning from the objective structured clinical examination. Medical Education, 24, 219–223.

    Article  Google Scholar 

  • Shen, L. (1994). Gender effects on student performances on the NBOME Part I, Part II, and Part III. Academic Medicine, 69(10), S75–S77. [Supplement].

    Article  Google Scholar 

  • Siu, E., & Reiter, H. I. (2009). Overview: What’s worked and what hasn’t as a guide towards predictive admissions tool development. Advances in Health Sciences Education, 14(5), 759–775. doi:10.1007/s10459-009-9160-8.

    Article  Google Scholar 

  • Tomayko, A., Roberts, W. L., Gimpel, J. R., & Boulet, J. R. (2008). Effect of examinee characteristics on success on a medical licensure examination. Paper presented at the American Educational Research Association, New York, NY.

  • Weidner, A. C., Gimpel, J. R., Boulet, J. R., & Solomon, M. (2010). Using standardized patients to assess the communication skills of graduating physicians for the comprehensive osteopathic medical licensing examination (COMLEX) Level 2-performance evaluation (Level 2-PE). Teaching and Learning in Medicine, 22(1), 8–15. doi:10.1080/10401330903445604.

    Article  Google Scholar 

  • Willms, J. D. (1992). Monitoring school performance: A guide for educators. Washington, DC: Falmer Press.

    Google Scholar 

Download references

Acknowledgments

We gratefully acknowledge the National Board of Osteopathic Medical Examiners, Inc. and American Association of Colleges of Osteopathic Medicine for support of this study. We thank Tom Levitan from the AACOM for his help with the matriculant records.

Author information

Authors and Affiliations

  1. National Board of Osteopathic Medical Examiners, Inc, 101 West Elm Street, Suite 150, Conshohocken, PA, 19428-2004, USA

    William L. Roberts & Erik Langenau

  2. UPMC Cancer Centers Breast Program, Magee-Women’s Hospital of UPMC, 300 Halket Street Room I-410, Pittsburgh, PA, 15213, USA

    Gina Pugliano

  3. Foundation for Advancement of International Medical Education and Research, 3624 Market Street, Philadelphia, PA, 19104, USA

    John R. Boulet

Authors
  1. William L. Roberts
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gina Pugliano
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Erik Langenau
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. John R. Boulet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William L. Roberts.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Roberts, W.L., Pugliano, G., Langenau, E. et al. Modeling relationships between traditional preadmission measures and clinical skills performance on a medical licensure examination. Adv in Health Sci Educ 17, 403–417 (2012). https://doi.org/10.1007/s10459-011-9321-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-011-9321-4

Keywords

Search

Navigation