Psychometrika

, Volume 75, Issue 2, pp 249–271 | Cite as

Paradoxical Results and Item Bundles

Theory and Methods

Abstract

Hooker, Finkelman, and Schwartzman (Psychometrika, 2009, in press) defined a paradoxical result as the attainment of a higher test score by changing answers from correct to incorrect and demonstrated that such results are unavoidable for maximum likelihood estimates in multidimensional item response theory. The potential for these results to occur leads to the undesirable possibility of a subject’s best answer being detrimental to them. This paper considers the existence of paradoxical results in tests composed of item bundles when compensatory models are used. We demonstrate that paradoxical results can occur when bundle effects are modeled as nuisance parameters for each subject. However, when these nuisance parameters are modeled as random effects, or used in a Bayesian analysis, it is possible to design tests comprised of many short bundles that avoid paradoxical results and we provide an algorithm for doing so. We also examine alternative models for handling dependence between item bundles and show that using fixed dependency effects is always guaranteed to avoid paradoxical results.

item response theory multidimensional item response theory likelihood paradoxical results item bundle random effects fixed dependency effects 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ackerman, T. (1996). Graphical representation of multidimensional item response theory analyses. Applied Psychological Measurement, 20(4), 311–329. CrossRefGoogle Scholar
  2. Bock, R., Gibbons, R., & Muraki, E. (1988). Full-information item factor analysis. Applied Psychological Measurement, 12, 261–280. CrossRefGoogle Scholar
  3. Craven, B.D. (1988). Fractional programming. Berlin: Heldermann. Google Scholar
  4. Douglas, J.A., Roussos, L.A., & Stout, W. (1996). Item-bundle dif hypothesis testing: Identifying suspect bundles and assessing their differential functioning. Journal of Educational Measurement, 33, 465–484. CrossRefGoogle Scholar
  5. Finkelman, M., Hooker, G., & Wang, J. (2009). Technical Report BU-1768-M, Department of Biological Statistics and Computational Biology, Cornell University. Google Scholar
  6. Hooker, G., Finkelman, M., & Schwartzman, A. (2009). Paradoxical results in multidimensional item response theory. Psychometrika, 74(3), 419–442. CrossRefGoogle Scholar
  7. Hoskens, M., & de Boeck, P. (1997). A parametric model for local dependence among test items. Psychological Methods, 2, 261–277. CrossRefGoogle Scholar
  8. Kelderman, H. (1984). Loglinear Rasch model tests. Psychometrika, 49, 223–245. CrossRefGoogle Scholar
  9. Li, Y., Bolt, D.M., & Fu, J. (2006). A comparison of alternative models for testlets. Applied Psychological Measurement, 20(1), 3–21. CrossRefGoogle Scholar
  10. McCullagh, P., & Nelder, J.A. (1989). Generalized linear models. London: Chapman and Hall/CRC. Google Scholar
  11. Reckase, M. (1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9, 401–412. CrossRefGoogle Scholar
  12. Rijmen, F., Tuerlinckx, F., de Boeck, P., & Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8(2), 185–205. CrossRefPubMedGoogle Scholar
  13. Rosenbaum, P.R. (1988). Item bundles. Psychometrika, 53, 349–359. CrossRefGoogle Scholar
  14. Veldkamp, B.P. (2002). Multidimensional constrained test assembly. Applied Psychological Measurement, 26(2), 133–146. CrossRefGoogle Scholar
  15. Wang, W., & Wilson, M. (2005). The Rasch testlet model. Applied Psychological Measurement, 29(2), 126–149. CrossRefGoogle Scholar
  16. Wang, X., Bradlow, E.T., & Wainer, H. (2002). A general Bayesian model for testlets: Theory and applications. Applied Psychological Measurement, 26(1), 109–128. CrossRefGoogle Scholar
  17. Wilson, M., & Adams, R.J. (1995). Rasch models for item bundles. Psychometrika, 60, 181–198. CrossRefGoogle Scholar

Copyright information

© The Psychometric Society 2009

Authors and Affiliations

  1. 1.Cornell UniversityIthacaUSA
  2. 2.Tufts School of Dental MedicineBostonUSA

Personalised recommendations