Abstract
Clinical skills assessments have traditionally been scored via experts' ratings of examinee performance. However, this approach to scoring may be impractical in a large-scale context due to logistical and cost considerations as well as the increased probability of rater error. The purpose of this investigation was therefore to identify, using discriminant analysis, weighted score-based models that maximize the accuracy with which mastery level can be estimated for examinees taking a nationally administered standardized patient test. Additionally, the accuracy with which the resulting classification functions can be applied to predict mastery level for a cross-validation sample of examinees was also examined. Results suggest that it might be feasible to implement an automated scoring procedure in a cost-effective manner while still retaining the important facets of the decision-making process of expert raters. Cost-benefit, test development and psychometric implications of these results are important and discussed in the full paper.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Baxter, G.P., Shavelson, R.J., Goldman, S.R. & Pine, J. (1992). Evaluation of procedure-based scoring for hands-on science assessment. Journal of Educational Measurement 29: 1-17.
Bejar, I.I. (1991). A methodology for scoring open-ended architectural design problems. Journal of Applied Psychology 76: 522-532.
Bennett, R.E. & Sebrechts, M.M. (1996). The accuracy of expert-system diagnoses of mathematical problem solutions. Applied Measurement in Education 9: 133-150.
Clauser, B.E., Subhiyah, R.G., Nungester, R.J., Ripkey, D.R., Clyman, S.G. & McKinley, D. (1995). Scoring a performance-based assessment by modeling the judgments of experts. Journal of Educational Measurement 32: 397-415.
Clauser, B.E., Margolis, M.J., Clyman, S.G. & Ross, L.P. (1997). Development of automated scoring algorithms for complex performance assessments: A comparison of two approaches. Journal of Educational Measurement 34: 141-161.
Hardy, R.A. (1995). Examining the costs of performance assessment. Applied Measurement in Education 8: 121-134.
Huberty, C.J. & Barton, R.M. (1989). An introduction to discriminant analysis. Measurement and Evaluation in Counseling and Development 22: 158-168.
Margolis, M.J., De Champlain, A.F. & Klass, D.J. (1998). Comparing Alternative Procedures for Scoring a Performance-Based Assessment of Physicians' Clinical Skills. Paper presented at the meeting of the National Council on Measurement in Education, San Diego, CA, April.
Margolis, M.J., De Champlain, A.F. & Klass, D.J. (1998). Setting examination-level standards for a performance-based assessment of physicians' clinical skills. Academic Medicine 73(10 suppl): S114-S116.
Norcini, J.J., Diserens, D., Day, S.C., Cebul, R.D., Schwartz, J.S., Beck, L.H., Webster, G.D., Schnabel, T.G. & Elstein, A. (1990). The scoring and reproducibility of an essay test of clinical judgment. Academic Medicine 65(9 suppl): S41-S42.
Thompson, B. (1998). Five Methodology Errors in Educational Research: The Pantheon of Statistical Significance and other Faux Pas. Paper presented at the meeting of the American Educational Research Association, San Diego, CA, April.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
De Champlain, A.F., Margolis, M.J., Macmillan, M.K. et al. Predicting Mastery Level on a Large-scale Standardized Patient Test: A Comparison of Case and Instrument Score-based Models Using Discriminant Function Analysis. Adv Health Sci Educ Theory Pract 6, 151–158 (2001). https://doi.org/10.1023/A:1011421706300
Issue Date:
DOI: https://doi.org/10.1023/A:1011421706300