Advertisement

Psychometrika

, Volume 67, Issue 3, pp 387–398 | Cite as

Hypergeometric family and item overlap rates in computerized adaptive testing

  • Hua-Hua Chang
  • Jinming ZhangEmail author
Articles

Abstract

A computerized adaptive test (CAT) is usually administered to small groups of examinees at frequent time intervals. It is often the case that examinees who take the test earlier share information with examinees who will take the test later, thus increasing the risk that many items may become known. Item overlap rate for a group of examinees refers to the number of overlapping items encountered by these examinees divided by the test length. For a specific item pool, different item selection algorithms may yield different item overlap rates. An important issue in designing a good CAT item selection algorithm is to keep item overlap rate below a preset level. In doing so, it is important to investigate what the lowest rate could be for all possible item selection algorithms. In this paper we rigorously prove that if every item has an equal possibility to be selected from the pool in a fixed-length CAT, the number of overlapping items among any α randomly sampled examinees follows the hypergeometric distribution family for α ≥ 1. Thus, the expected values of the number of overlapping items among any randomly sampled α examinees can be calculated precisely. These values may serve as benchmarks in controlling item overlap rates for fixed-length adaptive tests.

Key words

computerized adaptive testing hypergeometric distribution item exposure rate item selection stratification Sympson-Hetter method item overlap rate test security 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bickel, P.J., & Doksum, K.A. (1977).Mathematical statistics. San Francisco, CA: Holden-Day.Google Scholar
  2. Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's ability. In F.M. Lord & M.R. Novick,Statistical theories of mental test scores (pp. 395–479). Reading, MA: Addison-Wesley.Google Scholar
  3. Chang, H., & Ying, Z. (1996). A global information approach to computerized adaptive testing.Applied Psychological Measurement, 20, 213–229.Google Scholar
  4. Chang, H., & Ying, Z. (1999). A-stratified multistage in computerized adaptive testing.Applied Psychological Measurement, 23, 211–222.CrossRefGoogle Scholar
  5. Chen, S., Ankenmann, R.D., & Spray, J.A. (1999, April). Exploring the relationship between item exposure rate and item overlap rate in computerized adaptive testing. Paper presented at the 1999 Annual Meeting of the National Council on Measurement in Education, Montreal, Canada.Google Scholar
  6. Davey, T., & Parshall, C. (1995, April).New algorithms for item selection and exposure control with computerized adaptive testing. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.Google Scholar
  7. Hulin, C.L., Drasgow, F., & Parsons, C.K. (1983).Item response theory: Application to psychological measurement. Homewood, IL: Dow Jones-Irwin.Google Scholar
  8. Lord, F.M. (1970). Some test theory for tailored testing. In W.H. Holzman (Ed.),Computer assisted instruction, testing, and guidance. New York, NY: Harper and Row.Google Scholar
  9. Lord, F.M. (1980).Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  10. McBride, J.R., & Martin, J.T. (1983). Reliability and validity of adaptive ability tests in a military setting. In D.J. Weiss (Ed.),New horizons in testing (pp. 223–226). New York, NY: Academic Press.Google Scholar
  11. Owen, Z.J. (1975). A Bayesian sequential procedure for quantal response in the context of adaptive mental testing.Journal of American Statistical Association, 70, 351–356.Google Scholar
  12. Stocking, M.L. (1994).Three practical issues for modern adaptive testing item pools (ETS Research Rep. No. 94-5). Princeton, NJ: Educational Testing Service.Google Scholar
  13. Stocking, M.L., & Lewis, C. (1995).A new method of controlling item exposure in computerized adaptive testing (ETS Research Rep. No. 95-25). Princeton, NJ: Educational Testing Service.Google Scholar
  14. Stocking, M.L., & Lewis, C. (1998). Controlling item exposure conditional on ability in computerized adaptive testing.Journal of Educational and Behavioral Statistics, 23, 57–75.Google Scholar
  15. Sympson, J.B., & Hetter, R.D. (1985, October). Controlling item-exposure rates in computerized adaptive testing.Proceedings of the 27th Annual Meeting of the Military Testing Association (pp. 973–977). San Diego, CA: Navy Personnel Research and Development Center.Google Scholar
  16. Thomasson, G.L. (1995, June). New item exposure control algorithms for computerized adaptive testing. Paper presented at the annual meeting of the Psychometric Society, Minneapolis, MN.Google Scholar
  17. van der Linden, W.J., & Reese, L.M. (1998). A model for optimal constrained adaptive testing.Applied Psychological Measurement, 22, 259–270.Google Scholar
  18. Wainer, H. (1990).Computerized adaptive testing: A primer. Hillsdale, NJ: Lawrence Erlbaum Association.Google Scholar
  19. Wainer, H. (2000). Rescuing computerized testing by breaking Zipf's law,Journal of Educational and Behavioral Statistics, 25, 203–224.Google Scholar
  20. Way, W.D. (1998). Protecting the integrity of computerized testing item pools.Educational Measurement: Issues and Practice, Winter, 17–27.Google Scholar
  21. Weiss, D.J. (1976). Adaptive testing research in Minnesota: Overview, recent results, and future directions. In C.L. Clark (Ed.),Proceedings of the First Conference on Computerized Adaptive Testing (pp. 24–35). Washington, DC: United States Civil Service Commission.Google Scholar

Copyright information

© The Psychometric Society 2002

Authors and Affiliations

  1. 1.University of Texas at AustinUSA
  2. 2.MS 02-T, Educational Testing ServicePrinceton

Personalised recommendations