Advertisement

Psychometrika

, Volume 76, Issue 1, pp 13–39 | Cite as

Kullback–Leibler Information and Its Applications in Multi-Dimensional Adaptive Testing

  • Chun Wang
  • Hua-Hua Chang
  • Keith A. Boughton
Article

Abstract

This paper first discusses the relationship between Kullback–Leibler information (KL) and Fisher information in the context of multi-dimensional item response theory and is further interpreted for the two-dimensional case, from a geometric perspective. This explication should allow for a better understanding of the various item selection methods in multi-dimensional adaptive tests (MAT) which are based on these two information measures. The KL information index (KI) method is then discussed and two theorems are derived to quantify the relationship between KI and item parameters. Due to the fact that most of the existing item selection algorithms for MAT bear severe computational complexity, which substantially lowers the applicability of MAT, two versions of simplified KL index (SKI), built from the analytical results, are proposed to mimic the behavior of KI, while reducing the overall computational intensity.

Keywords

Kullback–Leibler information Fisher information multi-dimensional adaptive testing 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Adams, R.J., Wilson, M., & Wang, W.C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21(1), 1–23. CrossRefGoogle Scholar
  2. Allen, D.D., Ni, P.S., & Haley, S.M. (2008). Efficiency and sensitivity of multidimensional computerized adaptive testing of pediatric physical functioning. Disability and Rehabilitation, 30(6), 479–484. PubMedCrossRefGoogle Scholar
  3. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F.M. Lord & M.R. Novick (Eds.), Statistical theories of mental test scores (pp. 379–479). Reading: Addison-Welsey. Google Scholar
  4. Bloxom, B.M., & Vale, C.D. (1987). Multidimensional adaptive testing: a procedure for sequential estimation of the posterior centroid and dispersion of theta. Paper presented at the meeting of the Psychometric Society, Montreal, Canada. Google Scholar
  5. Chang, H.H. (1996). The asymptotic posterior normality of the latent trait for polytomous IRT models. Psychometrika, 61(3), 445–463. CrossRefGoogle Scholar
  6. Chang, H.H., Qian, J.H., & Ying, Z.L. (2001). a-stratified multistage computerized adaptive testing with b blocking. Applied Psychological Measurement, 25(4), 333–341. CrossRefGoogle Scholar
  7. Chang, H.H., & Stout, W. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58(1), 37–52. CrossRefGoogle Scholar
  8. Chang, H.H., & Ying, Z.L. (1996). A global information approach to computerized adoptive testing. Applied Psychological Measurement, 20(3), 213–229. CrossRefGoogle Scholar
  9. Chang, H.H., & Ying, Z.L. (1999). a-stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23(3), 211–222. CrossRefGoogle Scholar
  10. Chang, H.H., & Ying, Z.L. (2008). To weight or not to weight? Balancing influence of initial items in adaptive testing. Psychometrika, 73(3), 441–450. CrossRefGoogle Scholar
  11. Chang, H.H., & Zhang, J.M. (2002). Hypergeometric family and item overlap rates in computerized adaptive testing. Psychometrika, 67(3), 387–398. CrossRefGoogle Scholar
  12. Cover, T., & Thomas, J. (1991). Elements of information theory. New York: Wiley. CrossRefGoogle Scholar
  13. Finkelman, M., Nering, M.L., & Roussos, L.A. (2009). A conditional exposure control method for multidimensional adaptive testing. Journal of Educational Measurement, 46(1), 84–103. CrossRefGoogle Scholar
  14. Hambleton, R.K., & Swaminathan, H. (1985). Item response theory: principles and applications. Boston: Kluwer/Nijhoff. Google Scholar
  15. Henson, R., & Douglas, J. (2005). Test construction for cognitive diagnosis. Applied Psychological Measurement, 29(4), 262–277. CrossRefGoogle Scholar
  16. Haley, S.M., Ni, P.S., Ludlow, L.H., & Fragala-Pinkham, M.A. (2006). Measurement precision and efficiency of multidimensional computer adaptive testing of physical functioning using the pediatric evaluation of disability inventory. Archives of Physical Medicine and Rehabilitation, 87(9), 1223–1229. PubMedCrossRefGoogle Scholar
  17. Lehmann, E.L. (1986). Testing statistical hypotheses (2nd ed.). New York: Wiley. Google Scholar
  18. Lehmann, E.L., & Casella, G. (1998). Theory of point estimation (2nd ed.). Berlin: Springer. Google Scholar
  19. Lehmann, E.L. (1999). Elements of large-sample theory. New York: Springer. CrossRefGoogle Scholar
  20. Li, Y.H., & Schafer, W.D. (2005). Trait parameter recovery using multidimensional computerized adaptive testing in reading and mathematics. Applied Psychological Measurement, 29(1), 3–25. CrossRefGoogle Scholar
  21. Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale: Erlbaum. Google Scholar
  22. Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading: Addison-Welsley. Google Scholar
  23. Luecht, R.M. (1996). Multidimensional computerized adaptive testing in a certification or licensure context. Applied Psychological Measurement, 20(4), 389–404. CrossRefGoogle Scholar
  24. Mulder, J., & van der Linden, W.J. (2009). Multidimensional adaptive testing with optimal design criteria for item selection. Psychometrika, 74(2), 273–296. PubMedCrossRefGoogle Scholar
  25. Owen, R.J. (1969). A Bayesian approach to tailored testing (Research Report 69-92). Princeton, NJ: Educational Testing Service. Google Scholar
  26. Owen, R.J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. Journal of the American Statistical Association, 70, 351–356. CrossRefGoogle Scholar
  27. Passos, V.L., Berger, M.P.F., & Tan, F.E.S. (2008). The d-optimality item selection criterion in the early stage of CAT: a study with the graded response model. Journal of Educational and Behavioral Statistics, 33(1), 88–110. CrossRefGoogle Scholar
  28. Reckase, M.D. (1985). The difficulty of test items that measure more than one ability. Applied Psychological Measurement, 9(4), 401–412. CrossRefGoogle Scholar
  29. Reckase, M.D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21(1), 25–36. CrossRefGoogle Scholar
  30. Reckase, M.D. (2009). Multidimensional item response theory. New York: Springer. CrossRefGoogle Scholar
  31. Reckase, M.D., & McKinley, R.L. (1991). The discrimination power of items that measure more than one dimension. Applied Psychological Measurement, 15(4), 361–373. CrossRefGoogle Scholar
  32. Renyi, A. (1961). On measurs of entropy and information. In Proceeding fourth Berkeley symposium on mathematical statististics and probability (Vol. 1, pp. 547–561). Google Scholar
  33. Renyi, A. (1970). Probability theory. Amsterdam: North-Holland. Google Scholar
  34. Segall, D.O. (1996). Multidimensional adaptive testing. Psychometrika, 61(2), 331–354. CrossRefGoogle Scholar
  35. Segall, D.O. (2001). General ability measurement: an application of multidimensional item response theory. Psychometrika, 66(1), 79–97. CrossRefGoogle Scholar
  36. Stocking, M.L. (1994). Three practical issues for modern adaptive testing item pools (Research Report 94-5). Princeton, NJ: Educational Testing Service. Google Scholar
  37. Stroud, A.H., & Secrest, D. (1966). Gaussian quadrature formulas. Englewood Cliffs: Prentice-Hall. Google Scholar
  38. Thissen, D., & Mislevy, R.J. (2000). Testing algorithms. In H. Wainer (Ed.), Computerized adaptive testing: a primer (pp. 101–133). Hillsdale: Erlbaum. Google Scholar
  39. van der Linden, W.J. (1996). Assembling tests for the measurement of multiple traits. Applied Psychological Measurement, 20, 373–388. CrossRefGoogle Scholar
  40. van der Linden, W.J. (1998). Bayesian item selection criteria for adaptive testing. Google Scholar
  41. van der Linden, W.J. (1999). Multidimensional adaptive testing with a minimum error-variance criterion. Journal of Educational and Behavioral Statistics, 24(4), 398–412. Google Scholar
  42. van Rijn, P.W., Eggen, T., Hemker, B.T., & Sanders, P.F. (2002). Evaluation of selection procedures for computerized adaptive testing with polytomous items. Applied Psychological Measurement, 26(4), 393–411. CrossRefGoogle Scholar
  43. Veerkamp, W.J.J., & Berger, M.P.F. (1997). Some new item selection criteria for adaptive testing. Journal of Educational and Behavioral Statistics, 22(2), 203–226. Google Scholar
  44. Veldkamp, B.P., & van Der Linden, W.J. (2002). Multidimensional adaptive testing with constraints on test content. Psychometrika, 67(4), 575–588. CrossRefGoogle Scholar
  45. Wang, C., & Chang, H. (2009). Kullback–Leibler information in multidimensional adaptive testing: theory and application. In D.J. Weiss (Ed.), Proceedings of the 2009 GMAC conference on computerized adaptive testing. Retrieved from www.psych.umn.edu/psylabs/CATCentral/. Google Scholar
  46. Wang, C., & Chang, H. (2010). Item selection in multidimensional computerized adaptive testing—the new application of Kullback–Leibler information. Paper presented at 2010 International Meeting of Psychometric Society, Athens, Georgia. Google Scholar
  47. Yi, Q., & Chang, H.H. (2003). alpha-Stratified CAT design with content blocking. British Journal of Mathematical & Statistical Psychology, 56, 359–378. CrossRefGoogle Scholar

Copyright information

© The Psychometric Society 2010

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of Illinois at Urbana-ChampaignChampaignUSA
  2. 2.CTB/McGraw-HillMontereyUSA

Personalised recommendations