Skip to main content
SpringerLink
Log in

The risks of thoroughness: Reliability and validity of global ratings and checklists in an OSCE

  • Published:
Advances in Health Sciences Education Aims and scope Submit manuscript

Abstract

Objective: To compare checklists against global ratings for student performance on each station in an OSCE without the confounder of the global rating scorer having first filled in the checklist. Method: Subjects were 96 medical students completing their pre-clinical studies, who took an 8 station clinical OSCE. 39 students were assessed with detailed performance checklists; 57 students went through the same stations but were assessed using only a single global rating per station. A subset of 39 students were assessed by two independent raters. Results: Inter-rater and inter-station reliability of the global rating was the same as for the checklist. Correlation with a concurrent multiple choice test was similar for both formats. Conclusion: The global rating was found to be as reliable as more traditional checklist scoring. A discussion of the validity of checklist and global scores suggests that global ratings may be superior.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Barrows, H.S., Norman, G.R., Neufeld, V.R. & Feightner, J.W. (1982). The Clinical Reasoning Process of Randomly Selected Physicians in General Medical Practice. Clinical and Investigative Medicine 5: 49–56.

    Google Scholar 

  • Blake, J.M., Norman, G.R., Keane, D.R., Mueller, C.B., Cunnington, J.P.W & Didyk, N. (1996) Introducing Progress Testing in McMaster University's Problem Based Medical Curriculum: Psychometric Properties and Effect on Learning. Academic Medicine 71: 1002–1007.

    Google Scholar 

  • Cohen, D.S., Colliver, J.A., Marcy, M.S., Fried, E.D. & Swartz, M.H. (1996). Psychometric Properties of a Standardized-Patient Checklist and Rating-Scale Form Used to Assess Interpersonal and Communication Skills. Academic Medicine (suppl.) 71: s87–89.

    Google Scholar 

  • Cohen, R., Rothman, A.I., Poldre, P. & Ross, J. (1991). Validity and Generalizability of Global Ratings in an Objective Structured Clinical Examination. Academic Medicine 66: 545–548.

    Google Scholar 

  • Harden, R.M. & Gleeson, F.A. (1979). Assessment of Clinical Competence Using an Objective Structured Clinical Examination (OSCE). Medical Education 13: 41–54.

    Google Scholar 

  • MacRae, H.M., Vu, N.V. & Graham, B. et al. (1995). Comparing Checklists and Databases with Physicians' Ratings as Measures of Students History and Physical Examination Skills. Academic Medicine 70: 313–317.

    Google Scholar 

  • Marshall, J. (1977). Assessment of Problem Solving Ability. Medical Education 11: 329–334.

    Google Scholar 

  • Miller, G.E. (1968). The Orthopedic Training Study. Journal of the American Medical Association 206: 601–606.

    Google Scholar 

  • Neufeld, V.R., Norman, G.R., Barrows, H.S. & Feightner, J.W. (1981). Clinical Problem-Solving by Medical Students: A Longitudinal and Cross-Sectional Analysis. Medical Education 15: 315–322.

    Google Scholar 

  • Norcini, J.J., Diserens, D., Day, S.C., Cebul, R.C., Schwartz, J.S., Beck, L.H., Webster, G.D., Schnabel, T.G. & Elstein, A.S. (1990). The Scoring and Reproducability of an Essay Test of Clinical Judgement. Academic Medicine 65(suppl.): S41-S42.

    Google Scholar 

  • van der Vleuten, C.P.M., Norman, G.R. & de Graaff, E.D. (1991). Pitfalls in the Pursuit of Objectivity: Issues of Reliability. Medical Education 25: 119–126.

    Google Scholar 

  • van Luijk, S.J. & van der Vleuten, C.P.M. (1992). A Comparison of Checklists and Rating Scales in Performance-Based Testing. In Hart, I.R., Harden, R.M. & Des Marchais, J. (eds.) Current Developments in Assessing Clinical Competence, 357–382. Can-Heal Publications: Montreal.

    Google Scholar 

  • van der Vleuten, C.P.M., van Luijk, S.J. & Beckers, H.J.M. (1989). A Written Test as an Alternative to Performance Testing. Medical Education 23: 97–107.

    Google Scholar 

  • van Luijk, S.J., van der Vleuten, C.P.M. & van Schelven, S.M. (1990). Observer and Student Opinions About Performance-Based Tests. In Bender, W., Hiemstra, R.J., Scherpbier, A.J.J.A. & Zwierstra, R.P. (eds.) Teaching and Assessing Clinical Competence, 497–502. Boekwerk Publ.: Groningen.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. McMaster University, HSC 3U2, 1200 Main Street West, L5N 3Z5, Hamilton, Ontario, Canada

    J. P. W. Cunnington, A. J. Neville & G. R. Norman

Authors
  1. J. P. W. Cunnington
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. A. J. Neville
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. G. R. Norman
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Cunnington, J.P.W., Neville, A.J. & Norman, G.R. The risks of thoroughness: Reliability and validity of global ratings and checklists in an OSCE. Adv Health Sci Educ 1, 227–233 (1996). https://doi.org/10.1007/BF00162920

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF00162920

Key words

Search

Navigation