Advertisement

Advances in Health Sciences Education

, Volume 24, Issue 1, pp 141–150 | Cite as

The optimal number of options for multiple-choice questions on high-stakes tests: application of a revised index for detecting nonfunctional distractors

  • Mark R. RaymondEmail author
  • Craig Stevens
  • S. Deniz Bucak
Article

Abstract

Research suggests that the three-option format is optimal for multiple choice questions (MCQs). This conclusion is supported by numerous studies showing that most distractors (i.e., incorrect answers) are selected by so few examinees that they are essentially nonfunctional. However, nearly all studies have defined a distractor as nonfunctional if it is selected by fewer than 5% of examinees. A limitation of this definition is that the proportion of examinees available to choose a distractor depends on overall item difficulty. This is especially problematic for mastery tests, which consist of items that most examinees are expected to answer correctly. Based on the traditional definition of nonfunctional, a five-option MCQ answered correctly by greater than 90% of examinees will be constrained to have only one functional distractor. The primary purpose of the present study was to evaluate an index of nonfunctional that is sensitive to item difficulty. A secondary purpose was to extend previous research by studying distractor functionality within the context of professionally-developed credentialing tests. Data were analyzed for 840 MCQs consisting of five options per item. Results based on the traditional definition of nonfunctional were consistent with previous research indicating that most MCQs had one or two functional distractors. In contrast, the newly proposed index indicated that nearly half (47.3%) of all items had three or four functional distractors. Implications for item and test development are discussed.

Keywords

Assessment Multiple-choice questions Test development Item-writing guidelines High-stakes testing 

Notes

Acknowledgements

The authors express their gratitude to NBME for supporting this research. However, the opinions expressed here are those of the authors and do not necessarily reflect the position of NBME or the United States Medical Licensing Examination.

Compliance with ethical standards

Conflict of interest

The authors have no conflicts of interest to report. After IRB review by the American Institutes of Research, it was determined that this research is exempt from IRB review and oversight.

References

  1. Abdulghani, H. M., Ahmad, F., Ponnamperuma, G. G., Khalil, M. S., & Aldrees, A. (2014). The relationship between non-functioning distractors and item difficulty of multiple choice questions: A descriptive analysis. Journal of Health Specialties, 2, 148–151.CrossRefGoogle Scholar
  2. Abdulghani, H. M., Irshad, M., Haque, S., Ahmad, T., Sattar, K., & Khalil, M. S. (2017). Effectiveness of longitudinal faculty development programs on MCQs items writing skills: A follow-up study. PLoS ONE, 12(10), e0185895.  https://doi.org/10.1371/journal.pone.0185895.CrossRefGoogle Scholar
  3. Abozaid, H., Park, Y. S., & Tekian, A. (2017). Peer review improves psychometric characteristics of multiple choice questions. Medical Teacher, 39, s50–s54.CrossRefGoogle Scholar
  4. Cizek, G. J., & O’Day, D. (1994). Further investigation of nonfunctioning options in multiple-choice test items. Educational and Psychological Measurement, 54(4), 861–872.CrossRefGoogle Scholar
  5. Delgado, A. R., & Prieto, G. (1998). Further evidence favoring three-option items in multiple-choice tests. European Journal of Psychological Assessment, 14(3), 197–201.CrossRefGoogle Scholar
  6. Edwards, B. D., Arthur, W., & Bruce, L. L. (2012). The 3-option format for knowledge and ability multiple-choice tests: A case for why it should be more commonly used in personnel testing. International Journal of Selection and Assessment, 20(1), 65–81.CrossRefGoogle Scholar
  7. Fisher, K. M., Wandersee, J. H., & Moody, D. E. (2000). Mapping biology knowledge. Boston, MA: Kluwer Academic.Google Scholar
  8. Gierl, M. J., Balut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87, 1082–1116.CrossRefGoogle Scholar
  9. Gierl, M. J., Lai, H., Pugh, D., Touchie, C., Boulais, A. P., & De Champlain, A. (2016). Evaluating the characteristics of generated multiple-choice test items. Applied Measurement in Education, 29(3), 196–210.CrossRefGoogle Scholar
  10. Grier, J. B. (1975). The number of alternatives for optimum test reliability. Journal of Educational Measurement, 12, 109–112.CrossRefGoogle Scholar
  11. Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and validating test items. New York, NY: Routledge.CrossRefGoogle Scholar
  12. Jozefowicz, R. F., Koeppen, B. M., Case, S., Galbraith, R., Swanson, D., & Glew, R. H. (2002). The quality of in-house medical school examinations. Academic Medicine, 77, 156–161.CrossRefGoogle Scholar
  13. Kilgour, J. M., & Tayyaba, S. (2016). An investigation into the optimal number of distractors in single-best answer exams. Advances in Health Sciences Education, 21, 571–585.CrossRefGoogle Scholar
  14. Lord, F. M. (1944). Reliability of multiple choice tests as a function of number of choices per item. Journal of Educational Psychology, 35, 175–180.CrossRefGoogle Scholar
  15. Pappenberg, M., & Musch, J. (2017). Of small beauties and large beasts: The quality of distractors on multiple-choice tests is more important than their quantity. Applied Measurement in Education, 30(4), 273–286.CrossRefGoogle Scholar
  16. Rodriguez, M. C. (2005). Three options are optimal for multiple-choice test items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3–13.CrossRefGoogle Scholar
  17. Rogausch, A., Hofer, R., & Krebs, R. (2010). Rarely selected distractors in high stakes medical multiple choice examinations and their recognition by item authors: A simulation and survey. BMC Medical Education, 10, 85.CrossRefGoogle Scholar
  18. Roid, G. H., & Haladyna, T. M. (1982). A technology for test-item writing. New York: Academic Press.Google Scholar
  19. Schneid, S. D., Armour, C., Park, Y. S., Yudkowsky, R., & Bordage, G. (2014). Reducing the number of options on multiple-choice questions: Response time, psychometrics and standard setting. Medical Education, 48(10), 1020–1027.CrossRefGoogle Scholar
  20. Smith, J. K. (1982). Converging on correct answers: A peculiarity of multiple-choice items. Journal of Educational Measurement, 19(3), 211–220.CrossRefGoogle Scholar
  21. Tarrant, M., Ware, J., & Mohammed, A. M. (2009). An assessment of functioning and nonfunctioning distractors in multiple-choice questions: A descriptive analysis. BMC Medical Education, 9, 40–47.CrossRefGoogle Scholar
  22. Tversky, A. (1964). On the optimal number of alternatives at a choice point. Journal of Mathematical Psychology, 1, 386–391.CrossRefGoogle Scholar
  23. Wakefield, J. A. (1958). Does the fifth choice strengthen a test item? Public Personnel Review, 19, 44–48.Google Scholar
  24. Wallach, P. M., Crespo, L. M., Holtzman, K. Z., Galbraith, R. M., & Swanson, D. B. (2006). Use of a committee review process to improve the quality of course examination. Academic Medicine, 77(2), 156–161.Google Scholar

Copyright information

© Springer Nature B.V. 2018

Authors and Affiliations

  • Mark R. Raymond
    • 1
    Email author
  • Craig Stevens
    • 1
  • S. Deniz Bucak
    • 1
  1. 1.National Board of Medical ExaminersPhiladelphiaUSA

Personalised recommendations