Abstract
The decision to pass or fail a medical student is a ‘high stakes’ one. The aim of this study is to introduce and demonstrate the feasibility and practicality of a new objective standard-setting method for determining the pass/fail cut-off score from borderline grades. Three methods for setting up pass/fail cut-off scores were compared: the Regression Method, the Borderline Group Method, and the new Objective Borderline Method (OBM). Using Year 5 students’ OSCE results from one medical school we established the pass/fail cut-off scores by the abovementioned three methods. The comparison indicated that the pass/fail cut-off scores generated by the OBM were similar to those generated by the more established methods (0.840 ≤ r ≤ 0.998; p < .0001). Based on theoretical and empirical analysis, we suggest that the OBM has advantages over existing methods in that it combines objectivity, realism, robust empirical basis and, no less importantly, is simple to use.
Similar content being viewed by others
References
Behuniak, P., Archambault, F. X., & Gable, R. K. (1982). Angoff and Nedelsky standard setting procedures: Implications for the validity of proficiency test score interpretation. Educational and Psychological Measurement, 42(1), 247–255. doi:10.1177/0013164482421031.
Ben-David, M. (2000). AMEE Guide No. 18: Standard setting in student assessment. Medical Teacher, 22(2), 120–130.
Bowers, J., & Shindoll, R. (1989). A comparison of the angoff, beuk, and hofstee methods for setting a passing score (pp. 1–38). Iowa: ACT Organisation.
Cizek, G. J., & Bunch, M. (2006). Standard setting: A guide to establishing and evaluating performance standards on tests. London: Sage Pubns.
Cusimano, M. (1996). Standard setting in medical education. Academic Medicine, 71(10), S112–S120.
Ebel, R. L. (1972). Essentials of educational measurement. Englewood Cliffs, New Jersey: Prentice-Hall, Inc.
Halpin, G., & Halpin, G. (1987). An analysis of the reliability and validity of procedures for setting minimum competency standards. Educational and Psychological Measurement, 47, 977–983.
Harden, R. M., & Gleeson, F. A. (1979). Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education, 13(1), 39–54. doi:10.1111/j.1365-2923.1979.tb00918.x.
Hays, R., Sen Gupta, T., & Veitch, J. (2008). The practical value of the standard error of measurement in borderline pass/fail decisions. Medical Education, 42, 810–815.
Hurtz, G. M., & Auerbach, M. A. (2003). A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63(4), 584–601. doi:10.1177/0013164403251284.
Jalili, M., Hejri, S. M., & Norcini, J. J. (2011). Comparison of two methods of standard setting: The performance of the three-level Angoff method. Medical Education, 45(12), 1199–1208. doi:10.1111/j.1365-2923.2011.04073.x.
Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319–342. doi:10.1111/j.1745-3984.2001.tb01130.x.
Kramer, A., Muijtjens, A., Jansen, K., Düsman, H., Tan, L., & Van Der Vleuten, C. (2003). Comparison of a rational and an empirical standard setting procedure for an OSCE. Medical Education, 37(2), 132–139. doi:10.1046/j.1365-2923.2003.01429.x.
Livingston, S. A., & Zieky, M. J. (1982). Passing scores: Manual for setting standards of performance eonducational and occupational tests. Princeton, NJ: Educational Testing Service.
Messick, S. (1995a). Standards of validity and the validity of standards in performance asessment. Educational Measurement: Issues and Practice, 14(4), 5–8. doi:10.1111/j.1745-3992.1995.tb00881.x.
Messick, S. (1995b). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14(1), 3–19. doi:10.1177/001316445401400101.
Norcini, J. (2003). Setting standards on educational tests. Medical Education, 37(5), 464–469. doi:10.1046/j.1365-2923.2003.01495.x.
Norcini, J., & Shea, J. A. (1997). The credibility and comparability of standards. Applied Measurement in Education, 10(1), 39–59. doi:10.1207/s15324818ame1001_3.
Prideaux, D., Roberts, C., Eva, K., Centeno, A., Mccrorie, P., Mcmanus, C., et al. (2011). Assessment for selection for the health care professions and specialty training: Consensus statement and recommendations from the Ottawa 2010 Conference. Medical Teacher, 33, 215–223.
Schoonheim-Klein, M., Muijtjens, A., Habets, L., Manogue, M., van der Vleuten, C., & van der Velden, U. (2009). Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods. European Journal of Dental Education, 13(3), 162–171.
Schuwirth, L., Colliver, J., Gruppen, L., Kreiter, C., Mennin, S., Onishi, H., et al. (2011). Research in assessment: Consensus statement and recommendations from the Ottawa 2010 conference. Medical Teacher, 33(3), 224–233. doi:10.3109/0142159X.2011.551558.
Schuwirth, L., & van der Vleuten, C. P. M. (2012). Programmatic assessment and Kane’s validity perspective. Medical Education, 46(1), 38–48. doi:10.1111/j.1365-2923.2011.04098.x.
Wayne, D. B., Fudala, M. J., Butter, J., Siddall, V. J., Feinglass, J., Wade, L. D., et al. (2005). Comparison of two standard-setting methods for advanced cardiac life support training. Academic Medicine, 80(10), S63–S66.
Wilkinson, T., Newble, D., & Frampton, C. (2001). Standard setting in an objective structured clinical examination: Use of global ratings of borderline performance to determine the passing score. Medical Education, 35, 1043–1049.
Woehr, D. J., Arthur, W., & Fehrmann, M. L. (1991). An empirical comparison of cutoff score methods for content-related and criterion-related validity settings. Educational and Psychological Measurement, 51(4), 1029–1039. doi:10.1177/001316449105100423.
Wood, T., Humphrey-Murto, S., & Norman, G. (2006). Standard setting in a small scale OSCE: A comparison of the modified borderline-group method and the borderline regression method. Advances in Health Sciences Education, 11(2), 115–122. doi:10.1007/s10459-005-7853-1.
Zieky, M. J., & Livingston, S. A. (1977). Basic skills assessment. Manual for setting standards on the basic skills assessment tests. New Jersey Princeton, NJ: Educational Testing Service.
Author information
Authors and Affiliations
Corresponding author
Appendix 1: Practical instructions and an example
Appendix 1: Practical instructions and an example
How to use the OBM for setting up an objective pass/fail cut-off score.
The example below includes scores from OSCE station 1 used in the study. The borderline grades are defined as all scores between 6.75 and 7.25 inclusive, in total 12 scores.
The distribution of scores and classification for ‘Pass’ (P), ‘Borderline’ (B) and ‘Fail’ (F) is presented in Table 4. Overall there are 179 scores distributed as follow, where F, B, and P stand for the number of grades in each category:
Number of fails (F) | F = 22 |
Number of borderlines (B) | B = 12 |
Number of passes (P) | P = 145 |
Use Eq. 1 to identify the score which will be used as the pass/fail cut-off.
Equation 1 the quantile determining the cut-off score
Place F = 13, B = 12 and P = 145
The 67.4th percentile among the borderline scores determines the cut-off score.
Since there are 12 borderline (B) scores (see the shaded rows in Table 4), the cut-off score is the score above the 67.4 % of the borderline scores which is the 8.09th (12 × 0.674 = 8.09) or rounded up the 9th borderline score. In Table 4 we see that the 9th borderline score is 7.25 which is the minimum pass score for this OSCE examination for this population of examinees.
Note that six of the borderline grades have scores of 7.25 and it is impossible to rank them, hence everyone who achieved 7.25 or more will receive a pass grade in this station (similar to any other standard-setting method when a number of borderline scores could not be ranked).
Rights and permissions
About this article
Cite this article
Shulruf, B., Turner, R., Poole, P. et al. The Objective Borderline method (OBM): a probability-based model for setting up an objective pass/fail cut-off score in medical programme assessments. Adv in Health Sci Educ 18, 231–244 (2013). https://doi.org/10.1007/s10459-012-9367-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10459-012-9367-y