Advertisement

Consequences of ignoring nested data structure on item parameters in Rasch/1P-IRT model

  • Yasuo MiyazakiEmail author
  • Youngyun Chungbaek
  • Kevin O. Shropshire
  • Donald Hedeker
Original Paper
  • 15 Downloads

Abstract

The present study investigated the consequences of ignoring a nested data structure on the Rasch/one parameter item response theory model. Although most large-scale educational assessment data do exhibit a nested data structure, current practice often ignores such data structure and applies the standard Rasch/IRT models to conduct measurement analyses. We hypothesized that this practice would produce negative consequences on the item parameter estimates. Using simulation, we investigated this hypothesis by comparing the results from an incorrectly specified two level model which ignored the nested data structure to those from a correctly specified three-level hierarchical generalized linear model. Use of the incorrect two-level model did, in fact, result in negative consequences in estimating the standard errors, although the point estimates were unbiased and identical to the ones from the three-level analysis. A real data set from the IEA Civic education study in 1999 was used to illustrate the simulation results.

Keywords

Rasch models Item response theory Multilevel models Nested data Standard errors 

Notes

Acknowledgements

The authors thank Marry Norris, Kevin Krost, three anonymous reviewers, and Dr. Maomi Ueno for providing useful feedbacks on earlier drafts to improve the quality of the manuscript.

Compliance with ethical standards

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

References

  1. Adams RJ, Wilson MR, Wang W (1997a) The multidimensional random coefficients multinomial logit model. Appl Psychol Meas 21:1–23CrossRefGoogle Scholar
  2. Adams RJ, Wilson MR, Wu M (1997b) Multilevel item response models: an approach to errors in variables regression. J Educ Behav Stat 22(1):46–75CrossRefGoogle Scholar
  3. Adams RJ, Wu ML, Carstensen CH (2007) Application of multivariate Rasch models in international large-scale educational assessments. In: von Davier M, Carstensen CH (eds) Multivariate and mixture distribution Rasch models: extensions and applications. Springer, New York, pp 271–280CrossRefGoogle Scholar
  4. Aitkin M, Aitkin I (2011) Statistical modeling of the national assessment of educational progress. Springer, New YorkCrossRefzbMATHGoogle Scholar
  5. Bock RD, Zimowski MF (1997) Multigroup IRT. In: van der Linden WJ, Hambleton RK (eds) Handbook of modern item response theory. Springer, New York, pp 433–448CrossRefGoogle Scholar
  6. Brown TA (2015) Confirmatory factor analysis for applied research, 2nd edn. Guilford Press, New YorkGoogle Scholar
  7. Embretson SE, Reise SP (2000) Item response theory for psychologists. Lawrence Erlbaum Associates Inc., MahwahGoogle Scholar
  8. Fox JP (2004) Application of multilevel IRT modeling. Sch Eff Sch Improv Int J Res Policy Pract 15(3–4):261–280CrossRefGoogle Scholar
  9. Fox JP (2005) Multilevel IRT using dichotomous and polytomous response data. Br J Math Stat Psychol 58:145–172MathSciNetCrossRefGoogle Scholar
  10. Fox JP (2007) Multilevel IRT modeling in practice with the package mlirt. J Stat Softw 20(5):1–16MathSciNetCrossRefGoogle Scholar
  11. Fox JP, Glas CAW (2001) Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika 66(2):271–288MathSciNetCrossRefzbMATHGoogle Scholar
  12. Hambleton RK, Swaminathan H, Rogers HJ (1991) Fundamentals of item response theory. Sage, Newbury ParkGoogle Scholar
  13. Hedeker D, Melmelstein RJ, Flay BR (2006) Application of item response theory for intensive longitudinal data. In: Walls TA, Schafer JL (eds) Models for intensive longitudinal data. Oxford University Press, New York, pp 84–108CrossRefGoogle Scholar
  14. Hedges LV, Hedberg EC (2007) Intra class correlation values for planning group-randomized trials in education. Educ Eval Policy Anal 29(1):60–87CrossRefGoogle Scholar
  15. Hox JJ (2010) Multilevel analysis, 2nd edn. Routledge, New YorkCrossRefzbMATHGoogle Scholar
  16. International Association for the Evaluation of Educational Achievement (IEA) (1999) 1999 CivEd data (data file and code book). Retrieved from http://rms.iea-dpc.org/
  17. Kamata A (2001) Item analysis by the hierarchical generalized linear model. J Educ Meas 38:79–93CrossRefGoogle Scholar
  18. Kamata A, Bauer DJ, Miyazaki Y (2008) Multilevel measurement model. In: O’Connell AA, McCoach DB (eds) Multilevel modeling of educational data. Information Age Publishing, Charlotte, pp 345–388Google Scholar
  19. Kish L (1965) Survey sampling. Wiley, New YorkzbMATHGoogle Scholar
  20. Maier K (2001) A Rasch hierarchical measurement model. J Educ Behav Stat 26:307–330CrossRefGoogle Scholar
  21. Meulders M, Xie Y (2004) Person-by-item predictors. In: De Boeck P, Wilson M (eds) Explanatory item response models. Springer, New York, pp 231–240Google Scholar
  22. Moerbeek M (2004) The consequence of ignoring a level of nesting in multilevel analysis. Multivar Behav Res 39:129–149CrossRefGoogle Scholar
  23. Muthén LK, Muthén BO (1998–2015) Mplus user’s guide, 7th edn. Muthén & Muthén, Los AngelesGoogle Scholar
  24. Olson JF, Martin MO, Mullis IVS (eds) (2008) TIMSS 2007 technical report. TIMSS & PIRIS International Study Center, Boston College, Chestnut HillGoogle Scholar
  25. Opdenakker M-C, Van Damme J (2000) The importance of identifying levels in multilevel analysis: an illustration of the effects of ignoring the top or intermediate levels in school effectiveness research. Sch Eff Sch Improv 11:103–130CrossRefGoogle Scholar
  26. Phillips GW (2015) Impacts of design effects in large-scale district and state assessments. Appl Meas Educ 28:33–47CrossRefGoogle Scholar
  27. Pinheiro JC, Bates DM (1995) Approximations to the log-likelihood function in the non-linear mixed effects model. J Commun Graph Stat 4:12–35Google Scholar
  28. Rabe-Hesketh S, Skrondal A (2012) Multilevel and longitudinal modeling using Stata (3rd ed.). Volume II: categorical responses, counts, and survival. Stata Press, College StationzbMATHGoogle Scholar
  29. Rasch G (1960) Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research, CopenhagenGoogle Scholar
  30. Raudenbush SW, Bryk AS (2002) Hierarchical linear models: applications and data analysis methods, 2nd edn. Sage, Thousand OaksGoogle Scholar
  31. Raudenbush SW, Bryk AS, Cheong Y, Congdon RT, du Toit M (2011) Hierarchical linear & nonlinear modeling (version 7.0) [computer software and manual]. Scientific Software Inc., LincolnwoodGoogle Scholar
  32. SAS Institute Inc. (2011) SAS/STAT (version 9.3) (Computer software). SAS Institute Inc., CaryGoogle Scholar
  33. Schulz W, Sibberns H (2004) IEA civic education study: technical report. IEA, AmsterdamGoogle Scholar
  34. Snijders TAB, Bosker RJ (1999) Multilevel analysis: an introduction to basic and advanced multilevel modeling. Sage, Thousand OakszbMATHGoogle Scholar
  35. Ten Have TR, Kunselman AR, Tran L (1999) A comparison of mixed effects logistic regression models for binary response data with two nested levels of clustering. Stat Med 18(8):947–960CrossRefGoogle Scholar
  36. Thissen D (1982) Marginal maximum likelihood estimation for the one-parameter logistic model. Psychometrika 47:175–186CrossRefzbMATHGoogle Scholar
  37. US Department of Education (2001) Office of educational research and improvement. national center for education statistics. The NAEP 1998 technical report, NCES 2001-509, Allen NL, Donoghue JR, Schoeps TL (2001) National Center for Education Statistics, Washington, DCGoogle Scholar
  38. Van den Noortgate W, Opdenakker. MC, Onghena P (2005) The effects of ignoring a level in multilevel analysis. Sch Eff Sch Improv 16:281–303CrossRefGoogle Scholar
  39. Van der Leeden R, Meijer E, Busing FM (2008) Resampling multilevel models. In: de Leeuw J, Meijer E (eds) Handbook of multilevel analysis. Springer, New York, pp 401–433CrossRefGoogle Scholar
  40. Van Landeghem G, De Fraine B, Van Damme J (2005) The consequence of ignoring a level of multilevel analysis: a comment. Multivar Behav Res 40(4):423–434CrossRefGoogle Scholar
  41. Yosef M (2001) A comparison of alternative approximations to maximum likelihood estimation for hierarchical generalized linear models: the logistic-normal model case. Doctoral dissertation. Retrieved from Dissertation Abstracts International, UMI No. 3036776Google Scholar

Copyright information

© The Behaviormetric Society 2019

Authors and Affiliations

  1. 1.School of EducationVirginia Polytechnic Institute and State UniversityBlacksburgUSA
  2. 2.Biocomplexity Institute of Virginia TechBlacksburgUSA
  3. 3.University of North Carolina Systems OfficeChapel HillUSA
  4. 4.Department of Public Health SciencesThe University of ChicagoChicagoUSA

Personalised recommendations