Skip to main content
SpringerLink
Log in

New Item-Selection Methods for Balancing Test Efficiency Against Item-Bank Usage Efficiency in CD-CAT

  • Conference paper
Quantitative Psychology Research

Part of the book series: Springer Proceedings in Mathematics & Statistics ((PROMS,volume 89))

Abstract

Cognitive diagnostic computerized adaptive testing (CD-CAT) is a popular mode of online testing for cognitive diagnostic assessment (CDA). A key issue in CD-CAT programs is item-selection methods. Existing popular methods can achieve high measurement efficiencies but fail to yield balanced item-bank usage. Diagnostic tests often have low stakes, so item overexposure may not be a major concern. However, item underexposure leads to wasted time and money on item development, and high test overlap leads to intense practice effects, which in turn threaten test validity. The question is how to improve item-bank usage without sacrificing too much measurement precision (i.e., the correct recovery of knowledge states) in CD-CAT, which is the major purpose of this study. We have developed several item-selection methods that successfully meet this goal. In addition, we have investigated the Kullback–Leibler expected discrimination (KL-ED) method that considers only measurement precision except for item-bank usage.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Breithaupt K, Ariel AA, Hare DR (2010) Assembling an inventory of multistage adaptive testing systems. In: van der Linden WJ, Glas CAW (eds) Elements of adaptive testing. Springer, New York, pp 247–266

    Google Scholar 

  • Chang H-H, Zhang J (2002) Hypergeometric family and item overlap rates in computerized adaptive testing. Psychometrika 67(3):387–398

    Article  MATH  MathSciNet  Google Scholar 

  • Chang H-H, Ying Z-L (1996) A global information approach to computerized adaptive testing. Appl Psychol Meas 20(3):213–229

    Article  Google Scholar 

  • Chang H-H, Ying Z-L (1999) a-stratified multistage computerized adaptive testing. Appl Psychol Meas 23(3):211–222

    Google Scholar 

  • Chang Y-CI, Lu H-Y (2010) Online calibration via variable length computerized adaptive testing. Psychometrika 75(1):140–157

    Article  MATH  MathSciNet  Google Scholar 

  • Chen P, Xin T, Wang C, Chang H-H (2012) On-line calibration methods for the DINA model with independent attributes in CA-CAT. Psychometrika 77(2):201–222

    Article  MATH  MathSciNet  Google Scholar 

  • Cheng Y (2009) When cognitive diagnosis meets computerized adaptive testing: CD-CAT. Psychometrika 74(4):619–632

    Article  MATH  MathSciNet  Google Scholar 

  • Cheng Y (2010) Improving cognitive diagnostic computerized adaptive testing by balancing attribute coverage: the modified maximum global discrimination index method. Educ Psychol Meas 70(6):902–913

    Article  Google Scholar 

  • Collins JA, Greer JE, Huang SX (1993) Adaptive assessment using granularity hierarchies and Bayesian nets. Paper presented at the 3rd international conference intelligent tutoring systems

    Google Scholar 

  • de la Torre J, Douglas J (2004) Higher-order latent trait models for cognitive diagnosis. Psychometrika 69:333–353

    Article  MATH  MathSciNet  Google Scholar 

  • Embretson SE (1984) A general latent trait model for response processes. Psychometrika 49(2):175–186

    Article  MathSciNet  Google Scholar 

  • Haertel EH (1989) Using restricted latent class models to map the skill structure of achievement items. J Educ Meas 26(4):301–321

    Article  Google Scholar 

  • Henson R (2005) Test construction for cognitive diagnosis. Appl Psychol Meas 29(4):262–277

    Article  MathSciNet  Google Scholar 

  • Hsu CL, Wang WC, Chen SY (2013) Variable-length computerized adaptive testing based on cognitive diagnosis models. Appl Psychol Meas 37(7):563–582

    Article  Google Scholar 

  • Huebner A (2010) An overview of recent developments in cognitive diagnostic computer adaptive assessments. Pract Assess Res Eval 15(3):1–7. Available online: http://pareonline.net/getvn.asp?v=15&n=13

  • Junker BW, Sijtsma K (2001) Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Appl Psychol Meas 25:258–272

    Article  MathSciNet  Google Scholar 

  • Leighton JP, Gierl MJ (2007) Why cognitive diagnostic assessment? In: Leighton JP, Gierl MJ (eds) Cognitive diagnostic assessment for education: theory and applications. Cambridge University Press, New York, pp 3–18

    Chapter  Google Scholar 

  • Leighton JP, Gierl MJ, Hunka SM (2004) The attribute hierarchy method for cognitive assessment: a variation on Tatsuoka’s rule-space approach. J Educ Meas 41(3):205–237

    Article  Google Scholar 

  • Lin H-J, Ding S-L (2007) An exploration and realization of computerized adaptive testing with cognitive diagnosis. Acta Psychologica Sinica 39:747–753

    Google Scholar 

  • Liu H-Y, You X-F, Wang W-Y, Ding S-L, Chang H-H (2013) The development of computerized adaptive testing with cognitive diagnosis for an English achievement test in China. J Clas 30:152–172

    Google Scholar 

  • McGlohen MK (2004) The application of cognitive diagnosis and computerized adaptive testing to a large-scale assessment. Unpublished Doctorial Dissertation, University of Texas at Austin

    Google Scholar 

  • McGlohen MK, Chang H-H (2008) Combining computer adaptive testing technology with cognitively diagnostic assessment. Behav Res Methods 40(3):808–821

    Article  Google Scholar 

  • Millán E, Pérez-de-la-Cruz JL (2002) A Bayesian diagnostic algorithm for student modeling and its evaluation. User Model User-adapt Interact 12:281–330

    Article  MATH  Google Scholar 

  • Quellmalz ES, Pellegrino JW (2009) Technology and testing. Science 323(2):75–79

    Article  Google Scholar 

  • Representatives, U. S. H. o. (2001) Text of the ‘No Child Left Behind Act’. Public Law No. 107–110, 115 Stat. 1425

    Google Scholar 

  • Revuelta J, Ponsoda V (1998) A comparison of item exposure control methods in computerized adaptive testing. J Educ Meas 35(4):311–327

    Article  Google Scholar 

  • Shang Z-Y, Ding S-L (2011) The exploration of item selection strategy of computerized adaptive testing for cognitive diagnosis. J Jiangxi Norm Univ (Nat Sci) 35(4):418–421

    Google Scholar 

  • Tatsuoka C (2002) Data analytic methods for latent partially ordered classification models. J R Stat Soc: Ser C: Appl Stat 51:337–350

    Article  MATH  MathSciNet  Google Scholar 

  • Tatsuoka C, Ferguson T (2003) Sequential classification on partially ordered sets. J R Stat Soc Ser B (Stat Methodol) 65(1):143–157

    Article  MATH  MathSciNet  Google Scholar 

  • Tatsuoka KK (1995) Architecture of knowledge structures and cognitive diagnosis: a statistical pattern classification approach. In: Nichols PD, Chipman SF, Brennan RL (eds) Cognitively diagnostic assessments. Erlbaum, Hillsdale, pp 327–359

    Google Scholar 

  • Tatsuoka KK (2009) Cognitive assessment: an introduction to the rule space method. Taylor & Francis Group, New York

    Google Scholar 

  • Veldkamp BP, van der Linden WJ (2010) Designing item pools for adaptive testing. In: van der Linden WJ, Glas CAW (eds) Elements of adaptive testing. Springer, New York, pp 231–245

    Google Scholar 

  • Wang C (2013) Mutual information item selection method in cognitive diagnostic computerized adaptive testing with short test length. Educ Psychol Meas 73(6):1017–1035

    Article  Google Scholar 

  • Wang C, Chang H-H, Douglas J (2012) Combining CAT with cognitive diagnosis: a weighted item selection approach. Behav Res Methods 44:95–109

    Article  Google Scholar 

  • Wang C, Chang H-H, Huebner A (2011) Restrictive stochastic item selection methods in cognitive diagnostic computerized adaptive testing. J Educ Meas 48(3):255–273

    Article  Google Scholar 

  • Wu H-M, Kuo B-C, Yang J-M (2006) Evaluating knowledge structure-based adaptive testing algorithms and system development. Educ Technol Soc 15:73–88

    Google Scholar 

  • Xu XL, Chang HH, Douglas J (2003) A simulation study to compare CAT strategies for cognitive diagnosis. Paper presented at the annual meeting of the American Educational Research Association, Chicago

    Google Scholar 

Download references

Acknowledgments

This work is partially supported by the National Natural Science Foundation of China (30860084, 31160203, 31100756,31360237), the Ministry of Education of Humanities and Social Planning Project of China (13YJC880060), the Specialized Research Fund for the Doctoral Program of Higher Education (20103604110001, 20103604110002, 20113604110001), the Jiangxi Provincial Social Science Planning Project (12JY07), the Jiangxi Provincial Education Planning Project (13YB032), the Jiangxi Provincial Department of Education Science and Technology Project (GJJ11385, GJJ10238, GJJ13207, GJJ13226), and the Jiangxi Normal University Youth Growth Fund. All opinions and conclusions are solely those of the authors. The authors are indebted to the editor and reviewers for their constructive suggestions and comments on the earlier manuscript.

Author information

Authors and Affiliations

  1. College of Computer Information Engineering, Jiangxi Normal University, Nanchang, 330022, China

    Wenyi Wang & Shuliang Ding

  2. Elementary Educational College, Jiangxi Normal University, Nanchang, 330027, China

    Lihong Song

Authors
  1. Wenyi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuliang Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lihong Song
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lihong Song .

Editor information

Editors and Affiliations

  1. Department of Psychology, Arizona State University, Tempe, Arizona, USA

    Roger E. Millsap

  2. Dept. of Educational Psychology, University of Wisconsin, Madison, USA

    Daniel M. Bolt

  3. University of Amsterdam, Amsterdam, The Netherlands

    L. Andries van der Ark

  4. Department of Psychological Studies, The Hong Kong Institute of Education, Hong Kong, Hong Kong SAR

    Wen-Chung Wang

Rights and permissions

Reprints and permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Wang, W., Ding, S., Song, L. (2015). New Item-Selection Methods for Balancing Test Efficiency Against Item-Bank Usage Efficiency in CD-CAT. In: Millsap, R., Bolt, D., van der Ark, L., Wang, WC. (eds) Quantitative Psychology Research. Springer Proceedings in Mathematics & Statistics, vol 89. Springer, Cham. https://doi.org/10.1007/978-3-319-07503-7_8

Download citation

Publish with us

Policies and ethics

Search

Navigation