Skip to main content

Computerized adaptive testing under nonparametric IRT models

Abstract

Nonparametric item response models have been developed as alternatives to the relatively inflexible parametric item response models. An open question is whether it is possible and practical to administer computerized adaptive testing with nonparametric models. This paper explores the possibility of computerized adaptive testing when using nonparametric item response models. A central issue is that the derivatives of item characteristic Curves may not be estimated well, which eliminates the availability of the standard maximum Fisher information criterion. As alternatives, procedures based on Shannon entropy and Kullback–Leibler information are proposed. For a long test, these procedures, which do not require the derivatives of the item characteristic eurves, become equivalent to the maximum Fisher information criterion. A simulation study is conducted to study the behavior of these two procedures, compared with random item selection. The study shows that the procedures based on Shannon entropy and Kullback–Leibler information perform similarly in terms of root mean square error, and perform much better than random item selection. The study also shows that item exposure rates need to be addressed for these methods to be practical.

This is a preview of subscription content, access via your institution.

References

  • Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B.N. Petrov, & F. Csaki, (Eds.), Second international symposium on information theory (pp. 267–281); Budapest: Akadémiai Kiadó.

    Google Scholar 

  • Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F.M. Lord, & M.R. Novick (Eds.), Statistical theories of mental test scores (pp. 397–479) Reading, MA: Addison-Wesley.

    Google Scholar 

  • Chang, H.H.,& Stout, W.F. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37–52.

    Article  Google Scholar 

  • Chang, H.H., & Ying, Z. (1996). A global information approach to computerized adaptive testing. Applied Psychological Measurement, 20, 213–229.

    Google Scholar 

  • Cover, T.M., & Thomas, J.A. (1991). Elements of information theory. New York: Wiley.

    Google Scholar 

  • DeGroot M.H. (1962). Uncertainty, information and sequential experiments. Annals of Mathematical Statistics, 33, 404–419.

    Google Scholar 

  • Douglas, J. (1997). Joint consistency of nonparametric item characteristic curve and ability estimation. Psychometrika, 62, 7–28.

    Article  Google Scholar 

  • Eubank, R.L. (1988). Spline smoothing and nonparametric regression. New York, Marcel Dekker.

    Google Scholar 

  • Grayson, D.A. (1988). Two-group classification in latent trait theory: Scores with monotone likelihood ratio. Psychometrika, 53, 383–392.

    Article  Google Scholar 

  • He, X., & Ng, P. (1998). COBS: Qualitatively constrained smoothing via linear programming. Unpublished manual for SCOBS.

  • Nadaraya, E.A. (1964). On estimating regression. Probability Theory and its Applications, 9, 141–142.

    Article  Google Scholar 

  • Ramsay, J.O. (1991). Kernel smoothing approaches to nonparametric item characteristic curve estimation. Psychometrika, 56, 611–630.

    Article  Google Scholar 

  • Ramsay, J.O. (2000). TESTGRAF: A program for the graphical analysis of multiple choice test and questionnaire data [Computer Program]. Montreal: McGill University.

    Google Scholar 

  • Ramsay, J.O., & Abrahamowicz, M. (1989). Binomial regression with monotone splines: A psychometric application. Journal of the American Statistical Association, 84, 906–915.

    Article  Google Scholar 

  • Ramsay, J.O., & Winsberg, S. (1991). Maximum marginal likelihood estimation for semiparametric item analysis. Psychometrika, 56, 365–379.

    Article  Google Scholar 

  • Rossi, N., Wang, X., & Ramsay, J.O. (2002). Nonparametric item response function estimates with the EM algorithm. Journal of Educational and Behavioral Statistics, 27, 291–317.

    Google Scholar 

  • Shannon, C.E. (1948). A mathematical theory of communication, Bell Systems Techical Journal, 27, 379–423, 623–656.

    Google Scholar 

  • Tatsuoka, C. (2002). Data analytic methods for latent parially ordered classification models. Journal of the Royal Statistical Society, Series C, 51, 337–350.

    Article  Google Scholar 

  • Tatsuoka, C., & Ferguson, T. (2003). Sequential classification on patially ordered sets. Journal of Royal Statistical Society, Series B, 65, 143–158.

    Article  Google Scholar 

  • van der Linden, W.J., & Glas, C.A.W. (2000). Computerized adaptive testing: Theory and practice. Dordrecht: Kluwer Academic.

    Google Scholar 

  • Walker, A.M. (1969). On the asymptotic behavior of posterior distributions. Journal of the Royal Statistical Society, Series B, 31, 80–88.

    Google Scholar 

  • Watson, G.S. (1964). Smooth regression analysis. Sankhya, Series A, 26, 359–372.

    Google Scholar 

  • Xu X., Chang, H., & Douglas, J. (2003). A simulation study to compare CAT strategies for cognitive diagnosis. Presented at the Annual Meeting of the National Council of Measurement in Education, Chicago, April 2003.

Download references

Author information

Authors and Affiliations

Authors

Additional information

The authors would like to thank Hua Chang for his help in conducting this research.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Xu, X., Douglas, J. Computerized adaptive testing under nonparametric IRT models. Psychometrika 71, 121–137 (2006). https://doi.org/10.1007/s11336-003-1154-5

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-003-1154-5

Keywords

  • Shannon entropy
  • Kullback–Leibler information
  • nonparametric item response models
  • item response theory