Skip to main content

Item Analysis to Improve Reliability for an Internal Medicine Undergraduate OSCE

Advances in Health Sciences Education volume 10, pages 105–113 (2005)Cite this article

Abstract

Utilization of objective structured clinical examinations (OSCEs) for final assessment of medical students in Internal Medicine requires a representative sample of OSCE stations. The reliability and generalizability of OSCE scores provides validity evidence for OSCE scores and supports its contribution to the final clinical grade of medical students. The objective of this study was to perform item analysis using OSCE stations as the unit of analysis and evaluate the extent to which OSCE score reliability can be improved using item analysis data. OSCE scores from eight cohorts of fourth-year medical students (n = 435) in a 6-year undergraduate program were analyzed. Generalizability (G) coefficients of OSCE scores were computed for each cohort. Item analysis was performed by considering each OSCE station as an item and computing the corrected item-total correlation. OSCE stations which negatively impacted the reliability were deleted and the G-coefficient was recalculated. The G-coefficients of OSCE scores from the eight cohorts ranged from 0.48 to 0.80 (median 0.62). The median number of OSCE stations that negatively impacted the G-coefficient was 3.5 (out of a median of 25 total stations). When the ‘‘problem stations’’ were deleted, the median G-coefficient across eight cohorts increased to 0.62--0.72. In conclusion, item analysis of OSCE stations is useful and should be performed to improve the reliability of total OSCE scores. Problem stations can then be identified and improved.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

References

  • A. A-Latif (1992) ArticleTitleAn examination of the examinations: The reliability of the objective structured clinical examination Medical Teacher 14 179–183 Occurrence Handle1406127

    PubMed  Google Scholar 

  • J.R. Boulet D.W. McKinley G.P. Whelan R.K. Hambleton (2003) ArticleTitleQuality assurance methods for performance-based assessments Advances in Health Sciences Education 8 27–47 Occurrence Handle10.1023/A:1022639521218 Occurrence Handle12652167

    Article  PubMed  Google Scholar 

  • R.L. Brennan (2001) Generalizability Theory Springer-Verlag New York, NY

    Google Scholar 

  • J.P. Collins G.D. Gamble (1996) ArticleTitleA multi-format interdisciplinary final examination Medical Education 30 259–265 Occurrence Handle8949537

    PubMed  Google Scholar 

  • J.A Colliver R.G. Williams (1993) ArticleTitleTechnical issues: Test application Academic Medicine 68 454–458 Occurrence Handle8507310

    PubMed  Google Scholar 

  • J.A. Colliver S.J. Verhulst R.G. Williams J.J. Norcini (1989) ArticleTitleReliability of performance on standardized patient cases: A comparison of consistency measures based on generalizability theory Teaching and Learning in Medicine 1 31–37

    Google Scholar 

  • L.J. Cronbach G.C. Gleser H. Nanda N Rajaratnam (1972) The dependability of behavioral measurements: Generalizability for scores and profiles John Wiley and Sons New York

    Google Scholar 

  • J. Crossley H. Davies G. Humphries B. Jolly (2002) ArticleTitleGeneralisability: A key to unlock professional assessment Medical Education 36 972–978 Occurrence Handle10.1046/j.1365-2923.2002.01320.x Occurrence Handle12390466

    Article  PubMed  Google Scholar 

  • S.M. Downing (2003) ArticleTitleValidity: On the meaningful interpretation of assessment data Medical Education 37 830–837 Occurrence Handle10.1046/j.1365-2923.2003.01594.x Occurrence Handle14506816

    Article  PubMed  Google Scholar 

  • R. Harden M. Stevenson W. Downie G. Wilson (1975) ArticleTitleAssessment of clinical competence using objective structured examinations British Medical Journal 1 447–451 Occurrence Handle1115966

    PubMed  Google Scholar 

  • Kassam, N. (2003). Some validity evidence of an undergraduate Internal Medicine OSCE. Masters of Health Professions Education (MHPE) Thesis, University of Illinois at Chicago, Department of Medical Education, Chicago

  • D.G. Matsell N.M. Wolfish E. Hsu (1991) ArticleTitleReliability and validity of the objective structured clinical examination in pediatrics Medical Education 25 293–299 Occurrence Handle1890958

    PubMed  Google Scholar 

  • Messick, S.J. (1989). Validity. In R.L. Linn (ed), Educational Measurement (3rd ed), pp. 13--104. New York: American Council on Education and Macmillan

  • G.E. Miller (1990) ArticleTitleThe assessment of clinical skills/competence/performance Academic Medicine 65 S63–S67 Occurrence Handle2400509

    PubMed  Google Scholar 

  • D.I. Newble D.B. Swanson (1988) ArticleTitlePsychometric characteristics of the objective structured clinical examination Medical Education 22 325–334 Occurrence Handle3173161

    PubMed  Google Scholar 

  • G.R. Norman C.P.M. Van der Vleuten E. De Graff (1991) ArticleTitlePitfalls in the pursuit of objectivity: issues of validity efficiency and acceptability Medical Education 25 119–126 Occurrence Handle2023553

    PubMed  Google Scholar 

  • E.R. Petrusa T.A. Blackwell M.A. Ainsworth (1990) ArticleTitleReliability and validity of an objective structured clinical examination for assessing the clinical performance of residents Archives in Internal Medicine 150 573–577 Occurrence Handle10.1001/archinte.150.3.573

    Article  Google Scholar 

  • G. Regehr H. MacRae R.K. Reznick D. Szalay (1998) ArticleTitleComparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE format examination Academic Medicine 73 993–997 Occurrence Handle9759104

    PubMed  Google Scholar 

  • C. Van der Vleuten (2000) ArticleTitleValidity of final examinations in undergraduate medical training British Medical Journal 321 1217–1219 Occurrence Handle11073517

    PubMed  Google Scholar 

  • C.P.M. Van der Vleuten D.B. Swanson (1990) ArticleTitleAssessment of clinical skills with standardized patients: State of the art Teaching and Learning in Medicine 2 58–76

    Google Scholar 

  • BH. Verhoeven J. Hamers A.J. Scherpbier R.J. Hoogenboom C.P. Vleuten Particlevan der (2000) ArticleTitleThe effect on reliability of adding a separate written assessment component to an objective structured clinical examination Medical Education 34 525–529 Occurrence Handle10.1046/j.1365-2923.2000.00566.x Occurrence Handle10886634

    Article  PubMed  Google Scholar 

  • V. Wass C. Vleuten Particlevan der J. Shatzer R. Jones (2001) ArticleTitleAssessment of clinical competence Lancet 357 945–949 Occurrence Handle10.1016/S0140-6736(00)04221-5 Occurrence Handle11289364

    Article  PubMed  Google Scholar 

Download references

Author information

Affiliations

  1. Department of Medicine, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok, Thailand

    Chirayu Auewarakul & Rungnirand Praditsuwan

  2. Office of Medical Education, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok, Thailand

    Chirayu Auewarakul & Uapong Jaturatamrong

  3. Department of Medical Education, University of Illinois at Chicago, Chicago, USA

    Steven M. Downing

  4. Director of Medical Education Research Unit, Office of Medical Education, Faculty of Medicine Siriraj Hospital, 2 Prannok Road, Bangkoknoi, Bangkok, 10700, Thailand

    Chirayu Auewarakul

Authors
  1. Chirayu Auewarakul
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Steven M. Downing
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rungnirand Praditsuwan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Uapong Jaturatamrong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chirayu Auewarakul.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Auewarakul, C., Downing, S.M., Praditsuwan, R. et al. Item Analysis to Improve Reliability for an Internal Medicine Undergraduate OSCE. Adv Health Sci Educ Theory Pract 10, 105–113 (2005). https://doi.org/10.1007/s10459-005-2315-3

Download citation

  • Received:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-005-2315-3

Keywords

  • clinical competence
  • generalizability
  • item analysis
  • OSCE
  • performance assessment
  • reliability
  • undergraduate medical education