Abstract
For (0, 1) scored multiple-choice tests, a formula giving test reliability as a function of the number of item options is derived, assuming the “knowledge or random guessing model,” the parallelism of the new and old tests (apart from the guessing probability), and the assumptions of classical test theory. It is shown that the formula is a more general case of an equation by Lord, and reduces to Lord's equation if the items are effectively parallel. Further, the formula is shown to be closely related to another formula derived from Lord's randomly parallel tests model.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Carroll, J. B. (1945). The effect of difficulty and chance success on correlations between items or between tests.Psychometrika, X, 1–19.
Ebel, R. L. (1969). Expected reliability as a function of choices per item.Educational and Psychological Measurement, 29, 565–570.
Feldt, L.S. & Brennan, R.L. (1989). Reliability. In R.L. Linn (Ed.) Educational Measurement (3rd ed., pp. 105–147). New York: American Council on Education; Macmillan.
Grier, J. B. (1975). The number of alternatives for optimum test reliability.Journal of Educational Measurement, 12, 109–113.
Horst, P. (1954). The estimation of immediate retest reliability.Educational and Psychological Measurement, 14, 705–708.
Komaroff, E. (1997). Effect of simultaneous violations of essential tau-equivalence and uncorrelated error on Coefficient alpha.Applied Psychological Measurement, 21, 337–348.
Lord, F. M. (1944). Reliability of multiple choice tests as a function of choices per item.Journal of Educational Psychology, 35, 175–180.
Lord, F. M. (1957). Do tests of the same length have the same standard error of measurement?Educational and Psychological Measurement, 17, 510–521.
Lord, F. M. (1959). Tests of the same length do have the same standard error of measurement.Educational and Psychological Measurement, 19, 233–239.
Lord, F. M. (1977). Optimal number of choices per item—a comparison of four approaches.Journal of Educational Measurement, 14, 33–38.
Lord, F. M., & Novick, M. R. (1968).Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.
Mattson, D. (1965). The effects of guessing on the standard error of measurement and the reliability of test scores.Educational and Psychological Measurement, 25, 727–730.
Novick, M.R., & Lewis, C. (1967). Coefficient alpha and the reliability of composite measurements.Psychometrika, 32, 1–13.
Raykov, T. (2001). Estimation of congeneric scale reliability using covariance structure analysis with nonlinear constraints.British Journal of Mathematical and Statistical Psychology, 54, 315–323.
Trevisan, M.S., Sax, G., & Michael, W.B. (1994). Estimating the optimum number of options using an incremental option paradigm.Educational and Psychological Measurement, 54, 86–91.
Zimmerman, D.W. (1975). Probability spaces, Hilbert spaces, and the axioms of test theory.Psychometrika, 40, 395–412.
Zimmerman, D.W. (1976). Test theory with minimal assumptions.Educational and Psychological Measurement, 36, 85–96.
Zimmerman, D.W., Zumbo, B.D., & Lalonde, C. (1993). Coefficient alpha as an estimate of test reliability under violation of two assumptions.Educational and Psychological Measurement, 53, 33–49.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
MacCann, R.G. Reliability as a function of the number of item options derived from the “knowledge or random guessing” model. Psychometrika 69, 147–157 (2004). https://doi.org/10.1007/BF02295844
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF02295844