Advertisement

Utilizing Response Time in On-the-Fly Multistage Adaptive Testing

  • Yang DuEmail author
  • Anqi Li
  • Hua-Hua Chang
Conference paper
Part of the Springer Proceedings in Mathematics & Statistics book series (PROMS, volume 265)

Abstract

On-the-fly multistage adaptive testing (OMST), which integrates computerized adaptive testing (CAT) and multistage testing (MST), has recently gained popularity. While CAT selects each item on-the-fly and MST bundles items to pre-assembled modules, OMST assembles modules on-the-fly after the first stage. Since item selection algorithms play a crucial role in latent trait estimation and test security in CAT designs, given the availability of response time (RT) in the current testing era, researchers have been actively striving to incorporate RT into item selection algorithms. However, most such algorithms were only applied to CAT whereas little is known about RT’s role in the domain of OMST. Building upon previous research on RT-oriented item selection procedures, this research intends to apply RT-oriented item selection algorithms to OMST. This study found that the relative performance of RT-oriented item selection methods in OMST was consistent with CAT. But the underlying item bank structure and test design features can make a huge difference with respect to estimation accuracy and test security.

Keywords

On-the-fly multistage tests Response time CAT Test security Item bank usage 

Notes

Acknowledgements

The authors thank Dr. Dylan Molenaar for his detailed and helpful comments. Send request for reprints or further information to Yang Du at yangd2@illinois.edu or Anqi Li at anqili4@illinois.edu.

References

  1. Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6(4), 431–444.CrossRefGoogle Scholar
  2. Chang, H.-H., Qian, J., & Ying, Z. (2001). A-stratified multistage computerized adaptive testing with b blocking. Applied Psychological Measurement, 25(4), 333–341.MathSciNetCrossRefGoogle Scholar
  3. Chang, H.-H., & Ying, Z. (1999). A-stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23(3), 211–222.CrossRefGoogle Scholar
  4. Chen, S.-Y., Ankenmann, R. D., & Spray, J. A. (2003). The relationship between item exposure and test overlap in computerized adaptive testing. Journal of Educational Measurement, 40(2), 129–145.CrossRefGoogle Scholar
  5. Choe, E. M., Kern, J. L., & Chang, H.-H. (2018). Optimizing the use of response times for item selection in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 43, 135–158.CrossRefGoogle Scholar
  6. Fan, Z., Wang, C., Chang, H.-H., & Douglas, J. (2012). Utilizing response time distributions for item selection in cat. Journal of Educational and Behavioral Statistics, 37(5), 655–670.CrossRefGoogle Scholar
  7. Hau, K.-T., & Chang, H.-H. (2001). Item selection in computerized adaptive testing: Should more discriminating items be used first? Journal of Educational Measurement, 38(3), 249–266.CrossRefGoogle Scholar
  8. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.zbMATHGoogle Scholar
  9. Luecht, R. M. (2000). Implementing the computer-adaptive sequential testing (cast) framework to mass produce high quality computer-adaptive and mastery tests. Paper presented at the Annual Meeting of the National Council on Measurement in Education.Google Scholar
  10. van der Linden, W. J. (2006). A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics, 31(2), 181–204.CrossRefGoogle Scholar
  11. van der Linden, W. J., Breithaupt, K., Chuah, S. C., & Zhang, Y. (2007). Detecting differential speededness in multistage testing. Journal of Educational Measurement, 44(2), 117–130.CrossRefGoogle Scholar
  12. Wang, C., & Xu, G. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3), 456–477.MathSciNetCrossRefGoogle Scholar
  13. Yan, D., von Davier, A. A., & Lewis, C. (2014). Computerized multistage testing: Theory and applications. Taylor & Fransis Group.Google Scholar
  14. Zheng, Y., & Chang, H.-H. (2015). On-the-fly assembled multistage adaptive testing. Applied Psychological Measurement, 39(2), 104–118.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of Illinois, Urbana-ChampaignChampaignUSA
  2. 2.University of Illinois, Urbana-ChampaignChampaignUSA
  3. 3.Purdue UniversityWest LafayetteUSA

Personalised recommendations