Advances in Health Sciences Education

, Volume 10, Issue 2, pp 145–155 | Cite as

The Role of a Computerised Case-based Testing Procedure in Practice Performance Assessment

  • L. Schuwirth
  • S. Gorter
  • D. Van der. Heijde
  • J. J. Rethans
  • J. Brauer
  • H. Houben
  • S. Van der. Linden
  • C. Van der. Vleuten
  • A. Scherpbier
Article

Abstract

Introduction: For postgraduate training of doctors there is a need for valid and reliable instruments to assess their daily performance. Various instruments have been suggested, some of which use incognito simulated patients (SPs). These methods are resource intensive. Computerised Case-based testing (CCT) is logistically simpler and may still predict performance well. The research question was to evaluate the predictive validity of CCT for performance.

Methods: Seventeen rheumatologists were each visited by eight incognito SPs presenting various rheumatological complaints, and scoring the performance of the rheumatologists using a predefined checklist. From this checklist a panel of experts identified essential items. In addition the rheumatologists sat a CCT test containing 55 cases with a total of 121 items.

Results: Negative correlations were found between the SP scores and the CCT scores. This was unexpected. Therefore, background variables on experience were used to compare both methods. The correlation between these and CCT were high and positive and with the SP scores high and negative. This pattern did not differ when using the essential items of the checklist. Reliabilities of the SP scores were markedly high.

Discussion: Although CCT was not predictive of SP scores, it was related to working experience. There are good reasons to assume that although SP-scores were more authentic, they were less valid than CCT scores, mainly because they focussed more on thoroughness than on efficiency in data gathering. The results underpin the assumption that for valid performance assessment the most important issue is what information about the candidate is collected and now how authentic the method is.

Keywords

assessment authentic assessment incognito simulated patients performance assessment  rheumatology validity 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bordage G. (1987). An alternative approach to PMP’s: the “key-features” concept. In: Hart I.R., Harden R. (ed). Further Developments in Assessing Clinical Competence. Proceedings of the Second Ottawa Conference. Can-Heal Publications Inc, Montreal, pp. 59–75Google Scholar
  2. Bordage, G., Brailovsky, C., Carretier, H., Page, G. 1995Content validation of key features on a national examination of clinical decision-making skillsAcademic Medicine.70276281PubMedGoogle Scholar
  3. Chi, M.T.H., Glaser, R., Rees, E. 1982

    Expertise in problem solving

    Sternberg, R.J. eds. Advances in the Psychology of Human Intelligence.Lawrence Erlbaum AssociatesHillsdale, NJ776
    Google Scholar
  4. Davis, D.A., Thomson, M.A., Oxman, A.D., Haynes, R.B. 1995Changing physician performance. A systematic review of the effect of continuing medical education strategiesJAMA274700705CrossRefPubMedGoogle Scholar
  5. Glaser, R., Chi, M.T.H. 1988

    Overview

    Chi, M.T.H.Glaser, R.Farr, M.J. eds. The Nature of Expertise, xv–xxviii.Lawrence Erlbaum Associates Inc.Hillsdale, NJ, US
    Google Scholar
  6. Gorter, S., Linden, S., Brauer, J., Vander Heijde, D.H.H., Rethans, J.-J., Scherpbier, A., Vleuten, C., Boonen, A., Dinant, D., Goei The, H., Griep, E., Horst-Bruinsma, I., Linssen, A., Santen-Hoeufft, M., Tempel, H., Westgeest, T. 2001aRheumatologists’ performance in daily practiceArthritis Rheum451627CrossRefGoogle Scholar
  7. Gorter, S., Rethans, J.J., Scherpbier, A.J.J.A., Linden, S., Van Santen-Hoeufft, M.H.M., Vander Heijde, D.M.F.M., Houben, H.H.M.L., Vander Vleuten, C.P.M. 2001bHow to introduce incognito standardized patients into outpatient clinics of specialists in rheumatologyMedical Teacher23138144CrossRefGoogle Scholar
  8. Jansen, J.J., Tan, L.H., Vleuten, C.P., Luijk, S.J., Rethans, J.J., Grol, R.P. 1995Assessment of competence in technical clinical skills of general practitionersMedical Education.29247253PubMedGoogle Scholar
  9. Page G., Bordage G., Harasym P., Bowmer I., Swanson D.B. (1990). A new approach to assessing clinical problem-solving skills by written examination: Conceptual basis and initial pilot test results. In: Bender W., Hiemstra R.J., Scherpbier A., Zwierstra R.J. (ed). Teaching and Assessing Clinical Competence. Proceedings of the Fourth Ottawa Conference. Groningen: Boekwerk Publications, Groningen, The Netherlands, (403–407).Google Scholar
  10. Polsen, P., Jeffries, R. 1982

    Expertise in problem solving

    Sternberg, R.J. eds. Advances in the Psychology of Human Intelligence.Lawrence Erlbaum AssociatesHillsdale, NJ367411
    Google Scholar
  11. Posner, M.I. 1988

    What is it to be an expert?

    Chi, M.T.H.Glaser, R.Farr, M.J. eds. The Nature of Expertise, xxix–xxxvi.Lawrence Erlbaum Associates Inc.Hillsdale, NJ, US
    Google Scholar
  12. Ram, P. 1998Comprehensive assessment of general practitioners. Unpublished dissertationUniversity of MaastrichtMaastricht, The NetherlandsGoogle Scholar
  13. Regehr, G., Norman, G.R. 1996Issues in cognitive psychology: Implications for professional educationAcademic Medicine.719881001PubMedGoogle Scholar
  14. Rethans, J., Norcini, J., Báron-Maldonado, M., Blackmore, D., Jolly, B., LaDuca, T., Lew, S., Page, G., Southgate, L. 2002The relationship between competence and performance: implications for assessing practice performanceMedical Education.36901909CrossRefPubMedGoogle Scholar
  15. Rethans, J.J., Westin, S., Hays, R. 1996Methods for quality assessment in general practiceFamily Practice13468476PubMedGoogle Scholar
  16. Schuwirth, L., Southgate, L., Page, G., Paget, N., Lescop, J., Lew, S., Wade, W., Barón-Maldonado, M. 2002When enough is enough: A conceptual basis for fair and defensible practice performance assessmentMedical Education36925930CrossRefPubMedGoogle Scholar
  17. Schuwirth, L.W.T. 1998An approach to the assessment of medical problem solving: Computerised Case-based Testing (dissertation)University of MaastrichtMaastrichtGoogle Scholar
  18. Schuwirth, L.W.T., Blackmore, D.B., Mom, E.M.A, Vanden Wildenberg, F., Stoffers, H.E.J.M., Vander Vleuten, C.P.M. 1999How to write short cases for assessing problem-solving skillsMedical Teacher.21144150CrossRefGoogle Scholar
  19. Schuwirth, L.W.T., Vander Vleuten, C.P.M., De Kock, CA, Peperkamp, A.G.W., Donkers, H.H.L.M. 1996Computerized case-based testing: a modern method to assess clinical decision makingMedical Teacher.18295300Google Scholar
  20. Swanson, D.B. 1987

    A measurement framework for performance-based tests

    Hart, I.Harden, R. eds. Further developments in Assessing Clinical Competence.Can-Heal PublicationsMontreal1345
    Google Scholar
  21. Swanson, D.B., Norcini, J.J., Grosso, L.J. 1987Assessment of clinical competence: written and computer-based simulationsAssessment and Evaluation in Higher Education.12220246Google Scholar

Copyright information

© Springer 2005

Authors and Affiliations

  • L. Schuwirth
    • 1
    • 6
  • S. Gorter
    • 1
  • D. Van der. Heijde
    • 2
  • J. J. Rethans
    • 3
  • J. Brauer
    • 4
  • H. Houben
    • 4
  • S. Van der. Linden
    • 2
  • C. Van der. Vleuten
    • 1
  • A. Scherpbier
    • 5
  1. 1.Department of Educational Development and ResearchMaastricht UniversityThe Netherlands
  2. 2.Department of RheumatologyAcademic Hospital MaastrichtThe Netherlands
  3. 3.Skills laboratoryMaastricht UniversityThe Netherlands
  4. 4.Department of RheumatologyGeneral Hospital of HeerlenThe Netherlands
  5. 5.Educational bureau, Faculty of MedicineMaastricht UniversityThe Netherlands
  6. 6.Department of Educational Development and ResearchMaastricht UniversityThe Netherlands

Personalised recommendations