Skip to main content
Log in

Online Calibration Via Variable Length Computerized Adaptive Testing

  • Theory and Methods
  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

Item calibration is an essential issue in modern item response theory based psychological or educational testing. Due to the popularity of computerized adaptive testing, methods to efficiently calibrate new items have become more important than that in the time when paper and pencil test administration is the norm. There are many calibration processes being proposed and discussed from both theoretical and practical perspectives. Among them, the online calibration may be one of the most cost effective processes. In this paper, under a variable length computerized adaptive testing scenario, we integrate the methods of adaptive design, sequential estimation, and measurement error models to solve online item calibration problems. The proposed sequential estimate of item parameters is shown to be strongly consistent and asymptotically normally distributed with a prechosen accuracy. Numerical results show that the proposed method is very promising in terms of both estimation accuracy and efficiency. The results of using calibrated items to estimate the latent trait levels are also reported.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Abdelbasit, K.M., & Plackett, R.L. (1983). Experimental design for binary data. Journal of the American Statistical Association, 78, 90–98.

    Article  Google Scholar 

  • Baker, F.B. (1992). Item response theory: parameter estimation technique. New York: Marcel Dekker.

    Google Scholar 

  • Berger, M.P.F. (1992). Sequential sampling designs for the two-parameter item response theory model. Psychometrika, 57(4), 521–538.

    Article  Google Scholar 

  • Berger, M.P.F. (1994). D-optimal sequential sampling designs for item response theory models. Journal of Educational Statistics, 19(1), 43–56.

    Article  Google Scholar 

  • Berger, M.P.F., King, J., & Wong, W.K. (2000). Minimax d-optimal designs for item response theory models. Psychometrika, 65(3), 377–390.

    Article  Google Scholar 

  • Bock, R., & Aitken, M. (1981). Marginal maximum likelihood estimation of item parameters: an application of the em algorithm. Psychometrika, 46, 443–460.

    Article  Google Scholar 

  • Bock, R., & Mislevy, R. (1985). BILOG (Computer program). Scientific Software.

  • Chang, Y.-c.I. (1999). Strong consistency of maximum quasi-likelihood estimate in generalized linear models via a last time. Statistics and Probability Letters, 45, 237–246.

    Article  Google Scholar 

  • Chang, Y.-c.I. (2001). Sequential confidence regions of generalized linear models with adaptive designs. Journal of Statistical Planning and Inference, 93(1–2), 277.

    Article  Google Scholar 

  • Chang, Y.-c.I. (2006). Sequential estimation in generalized linear measurement-error models. Technical Report C-2006-13, Institute of Statistical Science, Academia Sinica.

  • Chang, Y.-c.I., & Martinsek, A. (1992). Fixed size confidence regions for parameters of a logistic regression model. Annals of Statistics, 20, 1953–1969.

    Article  Google Scholar 

  • Chang, Y.-c.I., & Ying, Z. (2004). Sequential estimate in variable length computerized adaptive testing. Journal of Statistical Planning and Inference, 121, 249–264.

    Article  Google Scholar 

  • Chen, K., Hu, I., & Ying, Z. (1999). Strong consistency of maximum quasi-likelihood estimators in generalized linear models with fixed and adaptive designs. Annals of Statistics, 27(4), 1155–1163.

    Article  Google Scholar 

  • Chow, Y.S., & Robbins, H. (1965). On the asymptotic theory of fixed-width sequential confidence intervals for the mean. Annals of Mathematical Statistics, 36(2), 457–462.

    Article  Google Scholar 

  • Chow, Y.S., & Teicher, H. (1997). Probability theory (3rd ed.). New York: Springer.

    Google Scholar 

  • Dmitrienko, A., & Govindarajulu, Z. (2000). Sequential confidence regions for maximum likelihood estimates. Annals of Statistics, 28(5), 1472–1501.

    Article  Google Scholar 

  • Fedorov, V.V. (1972). Theory of optimal design. New York: Academic Press.

    Google Scholar 

  • Ford, I. (1976). Optimal static and sequential design: a critical review. Doctoral Dissertation, University of Glasgow.

  • Ford, I., Titterington, D.M., & Kitsos, C.P. (1989). Recent advances in nonlinear experimental design. Technometrics, 31, 49–60.

    Article  Google Scholar 

  • Grambsch, P. (1983). Sequential sampling based on the observed fisher information. Annals of Statistics, 11, 68–77.

    Article  Google Scholar 

  • Grambsch, P. (1989). Sequential maximum likelihood estimation with applications to logistic regression in case-control studies. Journal of Statistical Planning and Inference, 22, 355–369.

    Article  Google Scholar 

  • Heise, M.A., & Myers, R.H. (1996). Optimal designs for bivariate logistic regression. Biometrics, 52, 613–624.

    Article  Google Scholar 

  • Jones, D.H., & Jin, Z. (1994). Optimal sequential designs for on-line item estimation. Psychometrika, 59, 59–75.

    Article  Google Scholar 

  • Kalish, L.A., & Rosenberger, J.L. (1978). Optimal designs for the estimation of the logistic function. Technical Report, Vol. 33, The Pennsylvania State University, Department of Statistics.

  • Khuri, A.I., Mukherjee, B., Sinha, B.K., & Ghosh, M. (2006). Design issues for generalized linear models, a review. Statistical Science, 21(3), 376–399.

    Article  Google Scholar 

  • Lai, T.L., & Wei, C.Z. (1982). Least squares estimates in stochastic regression models with applications to identification and control of dynamic systems. Annals of Statistics, 10, 154–166.

    Article  Google Scholar 

  • Li, T. (2002). Robust and consistent estimation of nonlinear errors-in-variables models. Journal of Econometrics, 110(1), 1–26.

    Article  Google Scholar 

  • Li, T., & Hsiao, C. (2004). Robust estimation of generalized linear models with measurement errors. Journal of Econometrics, 118(1–2), 51–65. Available at http://ideas.repec.org/a/eee/econom/v118y2004i1-2p51-65.html.

    Article  Google Scholar 

  • Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale: Lawrence Erlbaum Associates.

    Google Scholar 

  • Minkin, S. (1987). Optimal designs for binary data. Journal of the American Statistical Association, 82, 1098–1103.

    Article  Google Scholar 

  • Patz, R.J., & Junker, B.W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24, 146–178.

    Google Scholar 

  • Silvey, S.D. (1980). Optimal design. London: Chapman and Hall.

    Google Scholar 

  • Sitter, R.R. (1992). Robust designs for binary data. Biometrics, 48, 1145–1155.

    Article  Google Scholar 

  • Stefanski, L.A., & Carroll, R.J. (1985). Covariate measurement error in logistic regression. Annals of Statistics, 13(4), 1335–1351.

    Article  Google Scholar 

  • van der Linden, W.J., & Glas, C.A.W. (2000). Capitalization on item calibration error in adaptive testing. Applied Measurement in Education, 13(1), 35–53.

    Article  Google Scholar 

  • van der Linden, W.J., & Hambleton, R.K. (1997). Handbook of modern item response theory. Berlin: Springer.

    Google Scholar 

  • Wainer, H., & Mislevy, R. (2000). Computerized adaptive testing: a primer (2nd ed.). New Jersey: Lawrence Erlbaum Association.

    Google Scholar 

  • Wu, C.F.J. (1985). Efficient sequential designs with binary data. Journal of American Statistical Association, 80, 974–984.

    Article  Google Scholar 

  • Wynn, H.P. (1970). The sequential generation of d-optimum experimental designs. Annals of Mathematical Statistics, 41(5), 1655–1664.

    Article  Google Scholar 

  • Ying, Z., & Wu, C.J. (1997). An asymptotic theory of sequential designs based on maximum likelihood recursions. Statistica Sinica, 7, 75–91.

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Yuan-chin Ivan Chang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chang, Yc.I., Lu, HY. Online Calibration Via Variable Length Computerized Adaptive Testing. Psychometrika 75, 140–157 (2010). https://doi.org/10.1007/s11336-009-9133-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-009-9133-0

Keywords

Navigation