Multiple true–false items: a comparison of scoring algorithms
Multiple true–false (MTF) items are a widely used supplement to the commonly used single-best answer (Type A) multiple choice format. However, an optimal scoring algorithm for MTF items has not yet been established, as existing studies yielded conflicting results. Therefore, this study analyzes two questions: What is the optimal scoring algorithm for MTF items regarding reliability, difficulty index and item discrimination? How do the psychometric characteristics of different scoring algorithms compare to those of Type A questions used in the same exams? We used data from 37 medical exams conducted in 2015 (998 MTF and 2163 Type A items overall). Using repeated measures analyses of variance (rANOVA), we compared reliability, difficulty and item discrimination of different scoring algorithms for MTF with four answer options and Type A. Scoring algorithms for MTF were dichotomous scoring (DS) and two partial credit scoring algorithms, PS50 where examinees receive half a point if more than half of true/false ratings were marked correctly and one point if all were marked correctly, and PS1/n where examinees receive a quarter of a point for every correct true/false rating. The two partial scoring algorithms showed significantly higher reliabilities (αPS1/n = 0.75; αPS50 = 0.75; αDS = 0.70, αA = 0.72), which corresponds to fewer items needed for a reliability of 0.8 (nPS1/n = 74; nPS50 = 75; nDS = 103, nA = 87), and higher discrimination indices (rPS1/n = 0.33; rPS50 = 0.33; rDS = 0.30; rA = 0.28) than dichotomous scoring and Type A. Items scored with DS tend to be difficult (pDS = 0.50), whereas items scored with PS1/n become easy (pPS1/n = 0.82). PS50 and Type A cover the whole range, from easy to difficult items (pPS50 = 0.66; pA = 0.73). Partial credit scoring leads to better psychometric results than dichotomous scoring. PS50 covers the range from easy to difficult items better than PS1/n. Therefore, for scoring MTF, we suggest using PS50.
KeywordsAssessment Medical education Multiple choice Multiple true–false Scoring Undergraduates
The authors thank the examination board of the Swiss Federal Licensing Examination as well as the two Swiss medical schools for providing the data from the included exams. The authors wish to express their gratitude to the editor for the helpful guidance during the review process.
- Case, S. M., & Swanson, D. B. (2002). Constructing written test questions for the basic and clinical sciences (3rd ed.). Philadelphia, PA: National Board of Medical Examiners.Google Scholar
- Downing, S. M., & Yudkowsky, R. (2009). Assessment in health professions education. New York: Routledge.Google Scholar
- Dunham, M. L. (2006). An investigation of the multiple true–false item for nursing licensure and potential sources of construct-irrelevant difficulty. ProQuest.Google Scholar
- Itten, S., & Krebs, R. (1997). Messqualität der verschiedenen MC-Itemtypen in den beiden Vorprüfungen des Medizinstudiums an der Universität Bern 1997/2 (Forschungsbericht Institut für Aus-, Weiter-und Fortbildung (IAWF) der medizinischen Fakultät der Universität Bern). Bern: IAWF.Google Scholar
- Krebs, R. (2004). Anleitung zur Herstellung von MC-Fragen und MC-Prüfungen für die ärztliche Ausbildung. Bern: Institut für Medizinische Lehre IML, Abteilung für Ausbildungs-und Examensforschung AAE.Google Scholar
- R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria. Retrieved from http://www.r-project.org/.
- Tarasowa, D., & Auer, S. (2013). Balanced scoring method for multiple-mark questions. Paper presented at the CSEDU.Google Scholar
- Wu, B. C. (2003). Scoring multiple true false items: A comparison of summed scores and response pattern scores at item and test levels. Retrieved from Eric: https://eric.ed.gov/?id=ED476148