Skip to main content
SpringerLink
Log in

Effect of first-encounter pretest on pass/fail rates of a clinical skills medical licensure examination

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Due to the high-stakes nature of medical exams it is prudent for test agencies to critically evaluate test data and control for potential threats to validity. For the typical multiple station performance assessments used in medicine, it may take time for examinees to become comfortable with the test format and administrative protocol. Since each examinee in the rotational sequence starts with a different task (e.g., simulated clinical encounter), those who are administered non-scored pretest material on their first station may have an advantage compared to those who are not. The purpose of this study is to investigate whether pass/fail rates are different across the sequence of pretest encounters administered during the testing day. First-time takers were grouped by the sequential order in which they were administered the pretest encounter. No statistically significant difference in fail rates was found between examinees who started with the pretest encounter and those who encountered the pretest encounter later in the sequence. Results indicate that current examination administration protocols do not present a threat to the validity of test score interpretations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

  • Colliver, J. A., Vu, N. V., Verhulst, S. J., & Barrows, H. S. (1991). Effect of position-within-sequence on case performance in a multiple-stations examination using standardized-patient cases. Evaluation & the Health Professions, 14, 343–355.

    Article  Google Scholar 

  • De Champlain, A. F., Margolis, M. J., Ross, L. P., Macmillan, M. K., & Klass, D. J. (1998). Setting test-level standards for a performance assessment of physicians’ clinical skills: A process investigation. Paper presented at the American Educational Research Association, San Diego, CA.

  • Gimpel, J. R., Boulet, J. R., & Errichetti, A. M. (2003). Evaluating the clinical skills of osteopathic medical students. The Journal of the American Osteopathic Association, 103, 267–279.

    Google Scholar 

  • Gispert, R., Rue, M., Roma, J., & Martinez-Carretero, J. M. (1999). Gender, sequence of cases and day effects on clinical skills assessment with standardized patients. Medical Education, 33, 499–503.

    Article  Google Scholar 

  • Margolis, M. J., Clauser, B. E., Swanson, D. B., & Boulet, J. R. (2003). Analysis of the relationship between score components on a standardized patient clinical skills examination. Academic Medicine, 78(10), S68–S71. (Supplement issue).

    Article  Google Scholar 

  • McKinley, D. W., & Boulet, J. R. (2004). The effects of task sequence on examinee performance. Teaching and Learning in Medicine, 16, 18–22.

    Article  Google Scholar 

  • Norcini, J. J., & McKinley, D. W. (2007). Assessment methods in medical education. Teaching and Teacher Education, 23, 239–250.

    Article  Google Scholar 

  • Ramineni, C., Harik, P., Margolis, M. J., Clauser, B. E., Swanson, D. B., & Dillion, G. F. (2007). Sequence effects in the United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills (CS) examination. Academic Medicine, 82, S101–S104.

    Article  Google Scholar 

  • Roberts, J., & Norman, G. (1990). Reliability and learning from the objective structured clinical examination. Medical Education, 24, 219–223.

    Article  Google Scholar 

  • Swanson, D. B., Clauser, B. E., & Case, S. M. (1999). Clinical skills assessment with standardized patients in high-stakes tests: A framework for thinking about score precision, equating, and security. Advances in Health Sciences Education, 4, 67–106.

    Article  Google Scholar 

  • Whelan, G. P., Boulet, J. R., McKinley, D. W., Norcini, J. J., van Zanten, M., Hambelton, R. K., et al. (2005). Scoring standardized patient examinations: Lessons learned from the development and administration of the ECFMG clinical skills assessment (CSA®). Medical Teacher, 27, 200–206.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. National Board of Osteopathic Medical Examiners, Inc, 101 West Elm Street, Suite 150, Conshohocken, PA, 19428-2004, USA

    William L. Roberts

  2. Foundation for Advancement of International Medical Education and Research, 3624 Market Street, Philadelphia, PA, 19104-2685, USA

    Danette W. Mckinley & John R. Boulet

Authors
  1. William L. Roberts
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Danette W. Mckinley
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John R. Boulet
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to William L. Roberts.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Roberts, W.L., Mckinley, D.W. & Boulet, J.R. Effect of first-encounter pretest on pass/fail rates of a clinical skills medical licensure examination. Adv in Health Sci Educ 15, 219–227 (2010). https://doi.org/10.1007/s10459-009-9193-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-009-9193-z

Keywords

Search

Navigation