Skip to main content

Modeling Omitted and Not-Reached Items in IRT Models

Psychometrika volume 82, pages 795–819 (2017)Cite this article

Abstract

Item nonresponse is a common problem in educational and psychological assessments. The probability of unplanned missing responses due to omitted and not-reached items may stochastically depend on unobserved variables such as missing responses or latent variables. In such cases, missingness cannot be ignored and needs to be considered in the model. Specifically, multidimensional IRT models, latent regression models, and multiple-group IRT models have been suggested for handling nonignorable missing responses in latent trait models. However, the suitability of the particular models with respect to omitted and not-reached items has rarely been addressed. Missingness is formalized by response indicators that are modeled jointly with the researcher’s target model. We will demonstrate that response indicators have different statistical properties depending on whether the items were omitted or not reached. The implications of these differences are used to derive a joint model for nonignorable missing responses with ability to appropriately account for both omitted and not-reached items. The performance of the model is demonstrated by means of a small simulation study.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3

References

  • Adams, R. J., Wilson, M., & Wang, W. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21(1), 1–23. doi:10.1177/0146621697211001.

    Article  Google Scholar 

  • Adams, R. J., Wilson, M., & Wu, M. (1997). Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22(1), 47–76.

    Article  Google Scholar 

  • Antal, T. (2007). On the latent regression model of item response theory. Research Report No. RR-07-12. Princeton, NJ: Educational Testing Service.

  • Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques (2nd ed.). Boca Raton, FL: CRC. doi:10.2307/2532822.

  • Blossfeld, H.-P., Roßbach, H.-G., & von Maurice, J. (2011). Education as a lifelong process—The German national educational panel study (NEPS) [Special issue]. In Zeitschrift für Erziehungswissenschaft, 14. Wiesbaden: Springer VS.

  • Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459. doi:10.1007/BF02293801.

    Article  Google Scholar 

  • Cai, L. (2010). Metropolis-Hastings Robbins–Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35(3), 307–335. doi:10.3102/1076998609353115.

    Article  Google Scholar 

  • Cai, L., & Thissen, D. (2015). Modern approaches to parameter estimation in item response theory. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment. New York: Routledge.

    Google Scholar 

  • Culbertson, M. (2011, April). Is it wrong? Handling missing responses in IRT. Speech presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.

  • De Boeck, P. (2008). Random item IRT models. Psychometrika, 73(4), 533–559.

    Article  Google Scholar 

  • DeMars, C. (2002). Incomplete data and item parameter estimates under JMLE and MML. Applied Measurement in Education, 15, 15–31. doi:10.1207/S15324818AME1501_02.

    Article  Google Scholar 

  • Diggle, P., & Kenward, M. G. (1994). Informative drop-out in longitudinal data analysis. Applied Statistics, 43(1), 49–93.

    Article  Google Scholar 

  • Enders, C. K. (2010). Applied missing data analysis. New York: The Guilford Press.

    Google Scholar 

  • Fisher, R. A. (1925). Theory of statistical estimation. Proceedings of the Cambridge Philosophical Society, 22, 700–725.

    Article  Google Scholar 

  • Frey, A., Hartig, J., & Rupp, A. A. (2009). An NCME instructional module on booklet designs in large-scale assessments of student achievement: Theory and practice. Educational Measurement: Issues and Practice, 28(3), 39–53. doi:10.1111/j.1745-3992.2009.00154.x.

    Article  Google Scholar 

  • Glas, C. A. W., & Pimentel, J. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68(6), 907–922. doi:10.1177/0013164408315262.

    Article  Google Scholar 

  • Glynn, R. J., Laird, N. M., & Rubin, D. B. (1986). Selection modeling versus mixture modeling with nonignorable nonresponses. In H. Wainer (Ed.), Drawing inferences from self-selected samples. New York: Springer. doi:10.1007/978-1-4612-4976-4_10.

  • Harwell, M. R., Baker, F. B., & Zwarts, M. (1988). Item parameter estimation via marginal maximum likelihood and an em algorithm: A didactic. Journal of Educational and Behavioral Statistics, 13(3), 243–271.

    Article  Google Scholar 

  • Heckman, J. (1976). The common structure of statistical models of truncation, sample selection and limited dependent variables and a simple estimator for such models. The Annals of Economic and Social Measurement, 5, 475–492.

    Google Scholar 

  • Heckman, J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–61. doi:10.2307/1912352.

    Article  Google Scholar 

  • Holman, R., & Glas, C. A. W. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58(1), 1–17. doi:10.1111/j.2044-8317.2005.tb00312.x.

    Article  PubMed  Google Scholar 

  • Hsu, Y. (2000). On the Bock–Aitkin procedure—From an EM algorithm perspective. Psychometrika, 65, 547–549. doi:10.1007/BF02296345.

    Article  Google Scholar 

  • Huisman, M. (2000). Imputation of missing item responses: Some simple techniques. Quality & Quantity, 34, 331–351. doi:10.1023/A:1004782230065.

    Article  Google Scholar 

  • Kiefer, T., Robitzsch, A., & Wu, M. (2015). Tam: Test analysis modules [Computer software manual]. R package version 1.14-0. Retrieved from http://CRAN.R-project.org/package=TAM.

  • Korobko, O. K., Glas, C. A. W., Bosker, R. J., & Luyten, J. W. (2008). Comparing the difficulty of examination subjects with item response theory. Journal of Educational Measurement, 45, 137–155. doi:10.1111/j.1745-3984.2007.00057.x.

    Article  Google Scholar 

  • Little, R. J. A. (1993). Pattern-mixture models for multivariate incomplete data. Journal of the American Statistical Association, 88, 125–134. doi:10.2307/2290705.

    Google Scholar 

  • Little, R. J. A. (2008). Selection and pattern-mixture models. In G. Fitzmaurice, M. Davidian, G. Verbeke, & G. Molenberghs (Eds.), Longitudinal data analysis (pp. 409–432). Boca Raton, FL: Chapman & Hall/CRC.

    Chapter  Google Scholar 

  • Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). New York: Wiley.

    Google Scholar 

  • Lord, F. M. (1974). Estimation of latent ability and item parameters when there are omitted responses. Psychometrika, 39, 247–264. doi:10.1007/BF02291471.

    Article  Google Scholar 

  • Lord, F. M. (1983). Maximum likelihood estimation of item response parameters when some responses are omitted. Psychometrika, 48, 477–482. doi:10.1007/BF02293689.

    Article  Google Scholar 

  • Ludlow, L. H., & O’Leary, M. (1999). Scoring omitted and not-reached items: Practical data analysis implications. Educational and Psychological Measurement, 59(4), 615–630. doi:10.1177/0013164499594004.

    Article  Google Scholar 

  • Mislevy, R. J. (1984). Estimating latent distributions. Psychometrika, 49(3), 359–381. doi:10.1007/BF02306026.

    Article  Google Scholar 

  • Mislevy, R. J. (1985). Estimation of latent group effects. Journal of the American Statistical Association, 80(392), 993–997. doi:10.1080/01621459.1985.10478215.

    Article  Google Scholar 

  • Mislevy, R. J. (1987). Exploiting auxiliary information about examinees in the estimation of item parameters. Applied Psychological Measurement, 11(1), 81–91. doi:10.1177/014662168701100106.

    Article  Google Scholar 

  • Mislevy, R. J. (1988). Exploiting auxiliary information about items in the estimation of Rasch item diffculty parameters. Applied Psychological Measurement, 12(3), 281–296. doi:10.1177/014662168801200306.

    Article  Google Scholar 

  • Mislevy, R. J., & Wu, P. K. (1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing. Research Report No. RR-96-30. Princeton, NJ: Educational Testing Service.

  • Molenberghs, G., Kenward, M. G., & Lesaffre, E. (1997). The analysis of longitudinal ordinal data with nonrandom drop-out. Biometrika, 84(1), 33–44.

    Article  Google Scholar 

  • Moustaki, I., & Knott, M. (2000). Weighting for item non-response in attitude scales by using latent variable models with covariates. Journal of the Royal Statistical Society, Series A, 163, 445–459. doi:10.1111/1467-985X.00177.

    Article  Google Scholar 

  • Muthén, B. O., & Muthén, L. K. (1998–2012). Mplus User’s Guide (Version 7) [Computer software manual]. Los Angeles, CA: Muthn and Muthn.

  • OECD. (2009). PISA 2009. Technical Report. Paris: OECD.

  • O’Muircheartaigh, C., & Moustaki, I. (1999). Symmetric pattern models: A latent variable approach to item non-response in attitudes scales. Journal of the Royal Statistic Society, 162, 177–194. doi:10.1111/1467-985X.00129.

    Article  Google Scholar 

  • Pohl, S., Gräfe, L., & Rose, N. (2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74, 423–452. doi:10.1177/0013164413504926.

    Article  Google Scholar 

  • Pohl, S., Haberkorn, K., Hardt, K., & Wiegand, E. (2012). NEPS technical report for reading ? scaling results of starting cohort 3 in fifth grade. NEPS Working Paper No. 15. Bamberg: Otto-Friedrich-Universitt, Nationales Bildungspanel. Retrieved from https://www.neps-data.de/de-de/projekt%C3%BCbersicht/publikationen/nepsworkingpapers.aspx.

  • Robitzsch, A., & Ldtke, O. (2015). Ein genereller Ansatz zur Modellierung fehlender Item Responses in IRT-Modellen [A general approach for modeling item nonresponses in IRT models]. Speech presented at the 12. Tagung der Fachgruppe Methoden und Evaluation, Jena, September 2015.

  • Rose, N. (2013). Item nonresponses in educational and psychological measurement. Doctoral Thesis, Friedrich-Schiller-University, Jena. Retrieved from http://d-nb.info/1036873145/34.

  • Rose, N. (2015). Commonalities and differences in IRT-based methods for nonignorable item-nonresponses. Psychological Test and Assessment Modeling, 57, 472–478.

    Google Scholar 

  • Rose, N., von Davier, M., & Xu, X. (2010). Modeling nonignorable missing data with IRT. Research Report No. RR-10-11. Princeton, NJ: Educational Testing Service.

  • Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. doi:10.1093/biomet/63.3.581.

    Article  Google Scholar 

  • Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall. doi:10.1201/9781439821862.

    Book  Google Scholar 

  • Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. doi:10.1037/1082-989X.7.2.147.

    Article  PubMed  Google Scholar 

  • Schilling, S., & Bock, R. D. (2005). High-dimensional maximum marginal likelihood item factor analysis by adaptive quadrature. Psychometrika, 70(3), 533–555. doi:10.1007/s11336-003-1141-x.

    Google Scholar 

  • Van der Linden, W. J., Veldkamp, B. P., & Carlson, J. E. (2004). Optimizing balanced incomplete block designs for educational assessments. Applied Psychological Measurement, 28(5), 317–331. doi:10.1177/0146621604264870.

    Article  Google Scholar 

  • von Davier, M. (2009). Some notes on the reinvention of latent structure models as diagnostic classification models. Measurement: Interdisciplinary Research and Perspectives, 7, 67–74.

    Google Scholar 

  • von Davier, M., DiBello, L., & Yamamoto, K. (2008). Reporting test outcomes using models for cognitive diagnosis. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 151–176). Cambridge, MA: Hogrefe & Huber.

    Google Scholar 

  • von Davier, M., & Sinharay, S. (2010). Stochastic approximation methods for latent regression item response models. Journal of Educational and Behavioral Statistics, 35(2), 174–193.

    Article  Google Scholar 

  • Winship, C., & Mare, R. (1992). Models for sample selection bias. Annual Review of Sociology, 327–350. doi:10.1146/annurev.so.18.080192.001551.

  • Wu, M. L., Adams, R. J., & Wilson, M. R. (1998). ACER ConQest: Generalised item response modelling software [Computer software manual]. Mebourne: Australia.

Download references

Author information

Authors and Affiliations

  1. Hector Research Institute of Education Sciences and Psychology, University of Tübingen, Europastrasse 6, 72072 , Tübingen, Germany

    Norman Rose

  2. Educational Testing Service, 660 Rosedale Road, Princeton, NJ, 08541, USA

    Matthias von Davier

  3. Hector Research Institute of Education Sciences and Psychology, University of Tübingen, Europastrasse 6, 72072, Tübingen, Germany

    Benjamin Nagengast

Authors
  1. Norman Rose
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matthias von Davier
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Benjamin Nagengast
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Norman Rose.

Additional information

Parts of this paper are based on the unpublished dissertation of the first author. We thank Andreas Frey and Rolf Steyer who served as members on the thesis committee. We thank the anonymous reviewers for their careful reading of our manuscript and their many insightful comments and suggestions.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (docx 67 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rose, N., von Davier, M. & Nagengast, B. Modeling Omitted and Not-Reached Items in IRT Models. Psychometrika 82, 795–819 (2017). https://doi.org/10.1007/s11336-016-9544-7

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-016-9544-7

Keywords

  • nonignorable
  • item nonresponses
  • omitted and not-reached items
  • IRT