Advertisement

Psychometrika

, Volume 84, Issue 4, pp 1129–1151 | Cite as

The Asymptotic Distribution of Average Test Overlap Rate in Computerized Adaptive Testing

  • Edison M. ChoeEmail author
  • Hua-Hua Chang
Article
  • 128 Downloads

Abstract

The average test overlap rate is often computed and reported as a measure of test security risk or item pool utilization of a computerized adaptive test (CAT). Despite the prevalent use of this sample statistic in both literature and operations, its sampling distribution has never been known nor studied in earnest. In response, a proof is presented for the asymptotic distribution of a linear transformation of the average test overlap rate in fixed-length CAT. The theoretical results enable the estimation of standard error and construction of confidence intervals. Moreover, a practical simulation study demonstrates the statistical comparison of average test overlap rates between two CAT designs with different exposure control methods.

Keywords

test overlap item exposure test security pool utilization computerized adaptive testing asymptotic theory 

Notes

References

  1. Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques (2nd ed.). New York, NY: Marcel Dekker.CrossRefGoogle Scholar
  2. Barrada, J., Olea, J., Ponsoda, V., & Abad, F. J. (2009). Test overlap rate and item exposure as indicators of test security in CATs. In D. J. Weiss (Ed.), Proceedings of the 2009 GMAC conference on computerized adaptive testing.Google Scholar
  3. Chang, H.-H. (2015). Psychometrics behind computerized adaptive testing. Psychometrika, 80(1), 1–20.CrossRefGoogle Scholar
  4. Chang, H.-H., Qian, J., & Ying, Z. (2001). a-Stratified multistage computerized adaptive testing with b-blocking. Applied Psychological Measurement, 25(4), 333–341.CrossRefGoogle Scholar
  5. Chang, H.-H., & Ying, Z. (1999). a-Stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23(3), 211–222.CrossRefGoogle Scholar
  6. Chang, H.-H., & Ying, Z. (2009). Nonlinear sequential designs for logistic item response theory models with applications to computerized adaptive tests. The Annals of Statistics, 37(3), 1466–1488.CrossRefGoogle Scholar
  7. Chang, H.-H., & Zhang, J. (2002). Hypergeometric family and item overlap rates in computerized adaptive testing. Psychometrika, 67(3), 387–398.CrossRefGoogle Scholar
  8. Chen, S.-Y., Ankenmann, R. D., & Spray, J. A. (2003). The relationship between item exposure and test overlap in computerized adaptive testing. Journal of Educational Measurement, 40(2), 129–145.CrossRefGoogle Scholar
  9. Choe, E. M., Kern, J. L., & Chang, H.-H. (2018). Optimizing the use of response times for item selection in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 43(2), 135–158.CrossRefGoogle Scholar
  10. Cochran, W. G. (1977). Sampling techniques (3rd ed.). New York, NY: Wiley.Google Scholar
  11. Han, K. T. (2016). Maximum likelihood score estimation method with fences for short-length tests and computerized adaptive tests. Applied Psychological Measurement, 40(4), 289–301.CrossRefGoogle Scholar
  12. Hau, K.-T., & Chang, H.-H. (2001). Item selection in computerized adaptive testing: Should more discriminating items be used first? Journal of Educational Measurement, 38(3), 249–266.CrossRefGoogle Scholar
  13. Kingsbury, G. G., & Zara, A. R. (1989). Procedures for selecting items for computerized adaptive tests. Applied Measurement in Education, 2(4), 359–375.CrossRefGoogle Scholar
  14. Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.Google Scholar
  15. Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.Google Scholar
  16. Mislevy, R. J., & Chang, H.-H. (2000). Does adaptive testing violate local independence? Psychometrika, 65(2), 149–156.CrossRefGoogle Scholar
  17. Rencher, A. C., & Schaalje, G. B. (2008). Linear models in statistics (2nd ed.). Hoboken, NJ: Wiley-Interscience.Google Scholar
  18. Revuelta, J., & Ponsoda, V. (1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35(4), 311–327.CrossRefGoogle Scholar
  19. Sympson, J. B., & Hetter, R. D. (1985). Controlling item-exposure rates in computerized adaptive testing. In Proceedings of the 27th annual meeting of the Military Testing Association. San Diego, CA: Navy Personnel Research and Development Center.Google Scholar
  20. van der Linden, W. J., & Veldkamp, B. P. (2007). Conditional item-exposure control in adaptive testing using item-ineligibility probabilities. Journal of Educational and Behavioral Statistics, 32(4), 398–418.CrossRefGoogle Scholar
  21. Wang, C., Zheng, Y., & Chang, H.-H. (2014). Does standard deviation matter? Using “standard deviation” to quantify security of multistage testing. Psychometrika, 79(1), 154–174.CrossRefGoogle Scholar
  22. Way, W. D. (1998). Protecting the integrity of computerized testing item pools. Educational Measurement: Issues and Practice, 17, 17–27.CrossRefGoogle Scholar

Copyright information

© The Psychometric Society 2019

Authors and Affiliations

  1. 1.Graduate Management Admission Council™ (GMAC™)RestonUSA
  2. 2.Purdue UniversityWest LafayetteUSA

Personalised recommendations