Abstract
It has been widely reported that in computerized adaptive testing some examinees may get much lower scores than they would normally if an alternative paper-and-pencil version were given. The main purpose of this investigation is to quantitatively reveal the cause for the underestimation phenomenon. The logistic models, including the 1PL, 2PL, and 3PL models, are used to demonstrate our assertions. Our analytical derivation shows that, under the maximum information item selection strategy, if an examinee failed a few items at the beginning of the test, easy but more discriminating items are likely to be administered. Such items are ineffective to move the estimate close to the true θ, unless the test is sufficiently long or a variable-length test is used. Our results also indicate that a certain weighting mechanism is necessary to make the algorithm rely less on the items administered at the beginning of the test.
Similar content being viewed by others
References
Carlson, S. (2000). ETS finds flaws in the way online GRE rates some students. Chronicle of Higher Education, 47(8), A47.
Chang, H.H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213–229.
Chang, H.H., & Ying, Z. (1999). A-stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23(3), 211–222.
Hau, K.-T., & Chang, H. (2001). Item selection in computerized adaptive testing: should more discriminating items be used first? Journal of Educational Measurement, 28, 249–266.
Kaplan (2004). GRE Exam (2005 ed., p. 198). New York: Kaplan Publishing.
Lord, M.F. (1980). Applications of item response theory to practical testing problems. Hillsdale: Laurence Erlbaum Associates, Publishers.
Merritt, J. (2003). Why the folks at ETS flunked the course—a tech-savvy service will soon be giving B-school applicants their GMATs. Business Week, December 29, 2003.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was partially supported by the NSF Grants SES0241020 and SES0613025. The authors thank the Editor, Associate Editor and two anonymous reviewers for their comments and suggestions. Send further information to Hua-Hua Chang, Department of Psychology, 603 E. Daniel Street, M/C 716, Champaign, IL 61820.
Rights and permissions
About this article
Cite this article
Chang, HH., Ying, Z. To Weight or Not to Weight? Balancing Influence of Initial Items in Adaptive Testing. Psychometrika 73, 441–450 (2008). https://doi.org/10.1007/s11336-007-9047-7
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-007-9047-7