Abstract
Item nonresponse is a common problem in educational and psychological assessments. The probability of unplanned missing responses due to omitted and not-reached items may stochastically depend on unobserved variables such as missing responses or latent variables. In such cases, missingness cannot be ignored and needs to be considered in the model. Specifically, multidimensional IRT models, latent regression models, and multiple-group IRT models have been suggested for handling nonignorable missing responses in latent trait models. However, the suitability of the particular models with respect to omitted and not-reached items has rarely been addressed. Missingness is formalized by response indicators that are modeled jointly with the researcher’s target model. We will demonstrate that response indicators have different statistical properties depending on whether the items were omitted or not reached. The implications of these differences are used to derive a joint model for nonignorable missing responses with ability to appropriately account for both omitted and not-reached items. The performance of the model is demonstrated by means of a small simulation study.
Similar content being viewed by others
References
Adams, R. J., Wilson, M., & Wang, W. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21(1), 1–23. doi:10.1177/0146621697211001.
Adams, R. J., Wilson, M., & Wu, M. (1997). Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22(1), 47–76.
Antal, T. (2007). On the latent regression model of item response theory. Research Report No. RR-07-12. Princeton, NJ: Educational Testing Service.
Baker, F. B., & Kim, S.-H. (2004). Item response theory: Parameter estimation techniques (2nd ed.). Boca Raton, FL: CRC. doi:10.2307/2532822.
Blossfeld, H.-P., Roßbach, H.-G., & von Maurice, J. (2011). Education as a lifelong process—The German national educational panel study (NEPS) [Special issue]. In Zeitschrift für Erziehungswissenschaft, 14. Wiesbaden: Springer VS.
Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459. doi:10.1007/BF02293801.
Cai, L. (2010). Metropolis-Hastings Robbins–Monro algorithm for confirmatory item factor analysis. Journal of Educational and Behavioral Statistics, 35(3), 307–335. doi:10.3102/1076998609353115.
Cai, L., & Thissen, D. (2015). Modern approaches to parameter estimation in item response theory. In S. P. Reise & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment. New York: Routledge.
Culbertson, M. (2011, April). Is it wrong? Handling missing responses in IRT. Speech presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
De Boeck, P. (2008). Random item IRT models. Psychometrika, 73(4), 533–559.
DeMars, C. (2002). Incomplete data and item parameter estimates under JMLE and MML. Applied Measurement in Education, 15, 15–31. doi:10.1207/S15324818AME1501_02.
Diggle, P., & Kenward, M. G. (1994). Informative drop-out in longitudinal data analysis. Applied Statistics, 43(1), 49–93.
Enders, C. K. (2010). Applied missing data analysis. New York: The Guilford Press.
Fisher, R. A. (1925). Theory of statistical estimation. Proceedings of the Cambridge Philosophical Society, 22, 700–725.
Frey, A., Hartig, J., & Rupp, A. A. (2009). An NCME instructional module on booklet designs in large-scale assessments of student achievement: Theory and practice. Educational Measurement: Issues and Practice, 28(3), 39–53. doi:10.1111/j.1745-3992.2009.00154.x.
Glas, C. A. W., & Pimentel, J. L. (2008). Modeling nonignorable missing data in speeded tests. Educational and Psychological Measurement, 68(6), 907–922. doi:10.1177/0013164408315262.
Glynn, R. J., Laird, N. M., & Rubin, D. B. (1986). Selection modeling versus mixture modeling with nonignorable nonresponses. In H. Wainer (Ed.), Drawing inferences from self-selected samples. New York: Springer. doi:10.1007/978-1-4612-4976-4_10.
Harwell, M. R., Baker, F. B., & Zwarts, M. (1988). Item parameter estimation via marginal maximum likelihood and an em algorithm: A didactic. Journal of Educational and Behavioral Statistics, 13(3), 243–271.
Heckman, J. (1976). The common structure of statistical models of truncation, sample selection and limited dependent variables and a simple estimator for such models. The Annals of Economic and Social Measurement, 5, 475–492.
Heckman, J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–61. doi:10.2307/1912352.
Holman, R., & Glas, C. A. W. (2005). Modelling non-ignorable missing-data mechanisms with item response theory models. British Journal of Mathematical and Statistical Psychology, 58(1), 1–17. doi:10.1111/j.2044-8317.2005.tb00312.x.
Hsu, Y. (2000). On the Bock–Aitkin procedure—From an EM algorithm perspective. Psychometrika, 65, 547–549. doi:10.1007/BF02296345.
Huisman, M. (2000). Imputation of missing item responses: Some simple techniques. Quality & Quantity, 34, 331–351. doi:10.1023/A:1004782230065.
Kiefer, T., Robitzsch, A., & Wu, M. (2015). Tam: Test analysis modules [Computer software manual]. R package version 1.14-0. Retrieved from http://CRAN.R-project.org/package=TAM.
Korobko, O. K., Glas, C. A. W., Bosker, R. J., & Luyten, J. W. (2008). Comparing the difficulty of examination subjects with item response theory. Journal of Educational Measurement, 45, 137–155. doi:10.1111/j.1745-3984.2007.00057.x.
Little, R. J. A. (1993). Pattern-mixture models for multivariate incomplete data. Journal of the American Statistical Association, 88, 125–134. doi:10.2307/2290705.
Little, R. J. A. (2008). Selection and pattern-mixture models. In G. Fitzmaurice, M. Davidian, G. Verbeke, & G. Molenberghs (Eds.), Longitudinal data analysis (pp. 409–432). Boca Raton, FL: Chapman & Hall/CRC.
Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd ed.). New York: Wiley.
Lord, F. M. (1974). Estimation of latent ability and item parameters when there are omitted responses. Psychometrika, 39, 247–264. doi:10.1007/BF02291471.
Lord, F. M. (1983). Maximum likelihood estimation of item response parameters when some responses are omitted. Psychometrika, 48, 477–482. doi:10.1007/BF02293689.
Ludlow, L. H., & O’Leary, M. (1999). Scoring omitted and not-reached items: Practical data analysis implications. Educational and Psychological Measurement, 59(4), 615–630. doi:10.1177/0013164499594004.
Mislevy, R. J. (1984). Estimating latent distributions. Psychometrika, 49(3), 359–381. doi:10.1007/BF02306026.
Mislevy, R. J. (1985). Estimation of latent group effects. Journal of the American Statistical Association, 80(392), 993–997. doi:10.1080/01621459.1985.10478215.
Mislevy, R. J. (1987). Exploiting auxiliary information about examinees in the estimation of item parameters. Applied Psychological Measurement, 11(1), 81–91. doi:10.1177/014662168701100106.
Mislevy, R. J. (1988). Exploiting auxiliary information about items in the estimation of Rasch item diffculty parameters. Applied Psychological Measurement, 12(3), 281–296. doi:10.1177/014662168801200306.
Mislevy, R. J., & Wu, P. K. (1996). Missing responses and IRT ability estimation: Omits, choice, time limits, and adaptive testing. Research Report No. RR-96-30. Princeton, NJ: Educational Testing Service.
Molenberghs, G., Kenward, M. G., & Lesaffre, E. (1997). The analysis of longitudinal ordinal data with nonrandom drop-out. Biometrika, 84(1), 33–44.
Moustaki, I., & Knott, M. (2000). Weighting for item non-response in attitude scales by using latent variable models with covariates. Journal of the Royal Statistical Society, Series A, 163, 445–459. doi:10.1111/1467-985X.00177.
Muthén, B. O., & Muthén, L. K. (1998–2012). Mplus User’s Guide (Version 7) [Computer software manual]. Los Angeles, CA: Muthn and Muthn.
OECD. (2009). PISA 2009. Technical Report. Paris: OECD.
O’Muircheartaigh, C., & Moustaki, I. (1999). Symmetric pattern models: A latent variable approach to item non-response in attitudes scales. Journal of the Royal Statistic Society, 162, 177–194. doi:10.1111/1467-985X.00129.
Pohl, S., Gräfe, L., & Rose, N. (2014). Dealing with omitted and not-reached items in competence tests: Evaluating approaches accounting for missing responses in item response theory models. Educational and Psychological Measurement, 74, 423–452. doi:10.1177/0013164413504926.
Pohl, S., Haberkorn, K., Hardt, K., & Wiegand, E. (2012). NEPS technical report for reading ? scaling results of starting cohort 3 in fifth grade. NEPS Working Paper No. 15. Bamberg: Otto-Friedrich-Universitt, Nationales Bildungspanel. Retrieved from https://www.neps-data.de/de-de/projekt%C3%BCbersicht/publikationen/nepsworkingpapers.aspx.
Robitzsch, A., & Ldtke, O. (2015). Ein genereller Ansatz zur Modellierung fehlender Item Responses in IRT-Modellen [A general approach for modeling item nonresponses in IRT models]. Speech presented at the 12. Tagung der Fachgruppe Methoden und Evaluation, Jena, September 2015.
Rose, N. (2013). Item nonresponses in educational and psychological measurement. Doctoral Thesis, Friedrich-Schiller-University, Jena. Retrieved from http://d-nb.info/1036873145/34.
Rose, N. (2015). Commonalities and differences in IRT-based methods for nonignorable item-nonresponses. Psychological Test and Assessment Modeling, 57, 472–478.
Rose, N., von Davier, M., & Xu, X. (2010). Modeling nonignorable missing data with IRT. Research Report No. RR-10-11. Princeton, NJ: Educational Testing Service.
Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592. doi:10.1093/biomet/63.3.581.
Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall. doi:10.1201/9781439821862.
Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7(2), 147–177. doi:10.1037/1082-989X.7.2.147.
Schilling, S., & Bock, R. D. (2005). High-dimensional maximum marginal likelihood item factor analysis by adaptive quadrature. Psychometrika, 70(3), 533–555. doi:10.1007/s11336-003-1141-x.
Van der Linden, W. J., Veldkamp, B. P., & Carlson, J. E. (2004). Optimizing balanced incomplete block designs for educational assessments. Applied Psychological Measurement, 28(5), 317–331. doi:10.1177/0146621604264870.
von Davier, M. (2009). Some notes on the reinvention of latent structure models as diagnostic classification models. Measurement: Interdisciplinary Research and Perspectives, 7, 67–74.
von Davier, M., DiBello, L., & Yamamoto, K. (2008). Reporting test outcomes using models for cognitive diagnosis. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 151–176). Cambridge, MA: Hogrefe & Huber.
von Davier, M., & Sinharay, S. (2010). Stochastic approximation methods for latent regression item response models. Journal of Educational and Behavioral Statistics, 35(2), 174–193.
Winship, C., & Mare, R. (1992). Models for sample selection bias. Annual Review of Sociology, 327–350. doi:10.1146/annurev.so.18.080192.001551.
Wu, M. L., Adams, R. J., & Wilson, M. R. (1998). ACER ConQest: Generalised item response modelling software [Computer software manual]. Mebourne: Australia.
Author information
Authors and Affiliations
Corresponding author
Additional information
Parts of this paper are based on the unpublished dissertation of the first author. We thank Andreas Frey and Rolf Steyer who served as members on the thesis committee. We thank the anonymous reviewers for their careful reading of our manuscript and their many insightful comments and suggestions.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Rose, N., von Davier, M. & Nagengast, B. Modeling Omitted and Not-Reached Items in IRT Models. Psychometrika 82, 795–819 (2017). https://doi.org/10.1007/s11336-016-9544-7
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11336-016-9544-7