Skip to main content

The Objective Borderline method (OBM): a probability-based model for setting up an objective pass/fail cut-off score in medical programme assessments

Advances in Health Sciences Education volume 18, pages 231–244 (2013)Cite this article

Abstract

The decision to pass or fail a medical student is a ‘high stakes’ one. The aim of this study is to introduce and demonstrate the feasibility and practicality of a new objective standard-setting method for determining the pass/fail cut-off score from borderline grades. Three methods for setting up pass/fail cut-off scores were compared: the Regression Method, the Borderline Group Method, and the new Objective Borderline Method (OBM). Using Year 5 students’ OSCE results from one medical school we established the pass/fail cut-off scores by the abovementioned three methods. The comparison indicated that the pass/fail cut-off scores generated by the OBM were similar to those generated by the more established methods (0.840 ≤ r ≤ 0.998; p < .0001). Based on theoretical and empirical analysis, we suggest that the OBM has advantages over existing methods in that it combines objectivity, realism, robust empirical basis and, no less importantly, is simple to use.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2

References

  • Behuniak, P., Archambault, F. X., & Gable, R. K. (1982). Angoff and Nedelsky standard setting procedures: Implications for the validity of proficiency test score interpretation. Educational and Psychological Measurement, 42(1), 247–255. doi:10.1177/0013164482421031.

    Article  Google Scholar 

  • Ben-David, M. (2000). AMEE Guide No. 18: Standard setting in student assessment. Medical Teacher, 22(2), 120–130.

    Article  Google Scholar 

  • Bowers, J., & Shindoll, R. (1989). A comparison of the angoff, beuk, and hofstee methods for setting a passing score (pp. 1–38). Iowa: ACT Organisation.

    Google Scholar 

  • Cizek, G. J., & Bunch, M. (2006). Standard setting: A guide to establishing and evaluating performance standards on tests. London: Sage Pubns.

    Google Scholar 

  • Cusimano, M. (1996). Standard setting in medical education. Academic Medicine, 71(10), S112–S120.

    Article  Google Scholar 

  • Ebel, R. L. (1972). Essentials of educational measurement. Englewood Cliffs, New Jersey: Prentice-Hall, Inc.

    Google Scholar 

  • Halpin, G., & Halpin, G. (1987). An analysis of the reliability and validity of procedures for setting minimum competency standards. Educational and Psychological Measurement, 47, 977–983.

    Article  Google Scholar 

  • Harden, R. M., & Gleeson, F. A. (1979). Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Education, 13(1), 39–54. doi:10.1111/j.1365-2923.1979.tb00918.x.

    Article  Google Scholar 

  • Hays, R., Sen Gupta, T., & Veitch, J. (2008). The practical value of the standard error of measurement in borderline pass/fail decisions. Medical Education, 42, 810–815.

    Article  Google Scholar 

  • Hurtz, G. M., & Auerbach, M. A. (2003). A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus. Educational and Psychological Measurement, 63(4), 584–601. doi:10.1177/0013164403251284.

    Article  Google Scholar 

  • Jalili, M., Hejri, S. M., & Norcini, J. J. (2011). Comparison of two methods of standard setting: The performance of the three-level Angoff method. Medical Education, 45(12), 1199–1208. doi:10.1111/j.1365-2923.2011.04073.x.

    Article  Google Scholar 

  • Kane, M. T. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319–342. doi:10.1111/j.1745-3984.2001.tb01130.x.

    Article  Google Scholar 

  • Kramer, A., Muijtjens, A., Jansen, K., Düsman, H., Tan, L., & Van Der Vleuten, C. (2003). Comparison of a rational and an empirical standard setting procedure for an OSCE. Medical Education, 37(2), 132–139. doi:10.1046/j.1365-2923.2003.01429.x.

    Article  Google Scholar 

  • Livingston, S. A., & Zieky, M. J. (1982). Passing scores: Manual for setting standards of performance eonducational and occupational tests. Princeton, NJ: Educational Testing Service.

    Google Scholar 

  • Messick, S. (1995a). Standards of validity and the validity of standards in performance asessment. Educational Measurement: Issues and Practice, 14(4), 5–8. doi:10.1111/j.1745-3992.1995.tb00881.x.

    Article  Google Scholar 

  • Messick, S. (1995b). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.

    Article  Google Scholar 

  • Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14(1), 3–19. doi:10.1177/001316445401400101.

    Google Scholar 

  • Norcini, J. (2003). Setting standards on educational tests. Medical Education, 37(5), 464–469. doi:10.1046/j.1365-2923.2003.01495.x.

    Article  Google Scholar 

  • Norcini, J., & Shea, J. A. (1997). The credibility and comparability of standards. Applied Measurement in Education, 10(1), 39–59. doi:10.1207/s15324818ame1001_3.

    Article  Google Scholar 

  • Prideaux, D., Roberts, C., Eva, K., Centeno, A., Mccrorie, P., Mcmanus, C., et al. (2011). Assessment for selection for the health care professions and specialty training: Consensus statement and recommendations from the Ottawa 2010 Conference. Medical Teacher, 33, 215–223.

    Article  Google Scholar 

  • Schoonheim-Klein, M., Muijtjens, A., Habets, L., Manogue, M., van der Vleuten, C., & van der Velden, U. (2009). Who will pass the dental OSCE? Comparison of the Angoff and the borderline regression standard setting methods. European Journal of Dental Education, 13(3), 162–171.

    Article  Google Scholar 

  • Schuwirth, L., Colliver, J., Gruppen, L., Kreiter, C., Mennin, S., Onishi, H., et al. (2011). Research in assessment: Consensus statement and recommendations from the Ottawa 2010 conference. Medical Teacher, 33(3), 224–233. doi:10.3109/0142159X.2011.551558.

    Article  Google Scholar 

  • Schuwirth, L., & van der Vleuten, C. P. M. (2012). Programmatic assessment and Kane’s validity perspective. Medical Education, 46(1), 38–48. doi:10.1111/j.1365-2923.2011.04098.x.

    Article  Google Scholar 

  • Wayne, D. B., Fudala, M. J., Butter, J., Siddall, V. J., Feinglass, J., Wade, L. D., et al. (2005). Comparison of two standard-setting methods for advanced cardiac life support training. Academic Medicine, 80(10), S63–S66.

    Article  Google Scholar 

  • Wilkinson, T., Newble, D., & Frampton, C. (2001). Standard setting in an objective structured clinical examination: Use of global ratings of borderline performance to determine the passing score. Medical Education, 35, 1043–1049.

    Article  Google Scholar 

  • Woehr, D. J., Arthur, W., & Fehrmann, M. L. (1991). An empirical comparison of cutoff score methods for content-related and criterion-related validity settings. Educational and Psychological Measurement, 51(4), 1029–1039. doi:10.1177/001316449105100423.

    Article  Google Scholar 

  • Wood, T., Humphrey-Murto, S., & Norman, G. (2006). Standard setting in a small scale OSCE: A comparison of the modified borderline-group method and the borderline regression method. Advances in Health Sciences Education, 11(2), 115–122. doi:10.1007/s10459-005-7853-1.

    Article  Google Scholar 

  • Zieky, M. J., & Livingston, S. A. (1977). Basic skills assessment. Manual for setting standards on the basic skills assessment tests. New Jersey Princeton, NJ: Educational Testing Service.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centre for Medical and Health Sciences Education, Faculty of Medical and Health Sciences, University of Auckland, Private Bag 92019, Auckland Mail Centre, Auckland, 1142, New Zealand

    Boaz Shulruf

  2. Department of Statistics, Faculty of Sciences, University of Auckland, Private Bag 92019, Auckland Mail Centre, Auckland, 1142, New Zealand

    Rolf Turner

  3. Department of Medicine, Faculty of Medical and Health Sciences, University of Auckland, Private Bag 92019, Auckland Mail Centre, Auckland, 1142, New Zealand

    Phillippa Poole

  4. Health Care of the Elderly, University of Otago, The Princess Margaret Hospital, P.O. Box 800, Christchurch, New Zealand

    Tim Wilkinson

Authors
  1. Boaz Shulruf
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rolf Turner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Phillippa Poole
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tim Wilkinson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Boaz Shulruf.

Appendix 1: Practical instructions and an example

Appendix 1: Practical instructions and an example

How to use the OBM for setting up an objective pass/fail cut-off score.

The example below includes scores from OSCE station 1 used in the study. The borderline grades are defined as all scores between 6.75 and 7.25 inclusive, in total 12 scores.

The distribution of scores and classification for ‘Pass’ (P), ‘Borderline’ (B) and ‘Fail’ (F) is presented in Table 4. Overall there are 179 scores distributed as follow, where F, B, and P stand for the number of grades in each category:

Table 4 Example OSCE scores and grades (station 1)
Full size table

 

Number of fails (F) F = 22
Number of borderlines (B) B = 12
Number of passes (P) P = 145

Use Eq. 1 to identify the score which will be used as the pass/fail cut-off.

Equation 1 the quantile determining the cut-off score

$$ 1 - \left( {\frac{\text{B}}{{ ( {\text{B + F)}}}}} \right)*\left( {\frac{\text{P}}{{ ( {\text{P + B)}}}}} \right) $$
(1)

Place F = 13, B = 12 and P = 145

$$ 1 - \left( {\frac{12}{(12 + 22)}} \right)*\left( {\frac{145}{(145 + 12)}} \right) = 0.674 = 67.4\,\% $$

The 67.4th percentile among the borderline scores determines the cut-off score.

Since there are 12 borderline (B) scores (see the shaded rows in Table 4), the cut-off score is the score above the 67.4 % of the borderline scores which is the 8.09th (12 × 0.674 = 8.09) or rounded up the 9th borderline score. In Table 4 we see that the 9th borderline score is 7.25 which is the minimum pass score for this OSCE examination for this population of examinees.

Note that six of the borderline grades have scores of 7.25 and it is impossible to rank them, hence everyone who achieved 7.25 or more will receive a pass grade in this station (similar to any other standard-setting method when a number of borderline scores could not be ranked).

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Shulruf, B., Turner, R., Poole, P. et al. The Objective Borderline method (OBM): a probability-based model for setting up an objective pass/fail cut-off score in medical programme assessments. Adv in Health Sci Educ 18, 231–244 (2013). https://doi.org/10.1007/s10459-012-9367-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10459-012-9367-y

Keywords

  • Standard-setting
  • Borderline grades
  • Assessment
  • Cut-off score