Psychometrika

, Volume 76, Issue 2, pp 318–336

Measuring Growth in a Longitudinal Large-Scale Assessment with a General Latent Variable Model

  • Matthias von Davier
  • Xueli Xu
  • Claus H. Carstensen
Article

Abstract

The aim of the research presented here is the use of extensions of longitudinal item response theory (IRT) models in the analysis and comparison of group-specific growth in large-scale assessments of educational outcomes.

A general discrete latent variable model was used to specify and compare two types of multidimensional item-response-theory (MIRT) models for longitudinal data: (a) a model that handles repeated measurements as multiple, correlated variables over time and (b) a model that assumes one common variable over time and additional variables that quantify the change. Using extensions of these MIRT models, we approach the issue of modeling and comparing group-specific growth in observed and unobserved subpopulations. The analyses presented in this paper aim at answering the question whether academic growth is homogeneous across types of schools defined by academic demands and curricular differences. In order to facilitate answering this research question, (a) a model with a single two-dimensional ability distribution was compared to (b) a model assuming multiple populations with potentially different two-dimensional ability distributions based on type of school and to (c) a model that assumes that the observations are sampled from a discrete mixture of (unobserved) populations, allowing for differences across schools with respect to mixing proportions. For this purpose, we specified a hierarchical-mixture distribution variant of the two MIRT models. The latter model, (c), is a growth-mixture MIRT model that allows for variation of the mixing proportions across clusters in a hierarchically organized sample. We applied the proposed models to the PISA-I-Plus data for assessing learning and change across multiple subpopulations. The results of this study support the hypothesis of differential growth.

Keywords

item response theory growth models multidimensional IRT longitudinal models diagnostic models large-scale assessments 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. CrossRefGoogle Scholar
  2. Andersen, E.B. (1985). Estimating latent correlations between repeated testings. Psychometrika, 50, 3–16. CrossRefGoogle Scholar
  3. Andrade, D.F., & Tavares, H.R. (2005). Item response theory for longitudinal data: population parameter estimation. Journal of Multivariate Analysis, 95, 1–22. CrossRefGoogle Scholar
  4. Bock, R.D., & Zimowski, M.F. (1997). Multiple group IRT. In W.J. van der Linden, & R.K. Hambleton (Eds.), Handbook of modern item response theory (pp. 433–448). New York: Springer. Google Scholar
  5. Cai, L. (2010). High-dimensional exploratory item factor analysis by a Metropolis–Hastings Robbins–Monro algorithm. Psychometrika, 75, 33–57. CrossRefGoogle Scholar
  6. Draney, K., & Wilson, M. (2007). Application of the Saltus model to stage-like data: some applications and current developments. In M. von Davier, & C.H. Carstensen (Eds.), Multivariate and mixture distribution rasch models (pp. 119–130). New York: Springer. CrossRefGoogle Scholar
  7. Embretson, S.E. (1991). A multidimensional latent trait model for measuring learning and change. Psychometrika, 56, 495–515. CrossRefGoogle Scholar
  8. Embretson, S.E. (1997). Structured ability models in tests designed from cognitive theory. In M. Wilson, G. Engelhard, Jr., & K. Draney (Eds.), Objective measurement: theory into practice (Vol. 4, pp. 223–236). Greenwich: Ablex. Google Scholar
  9. Fischer, G.H. (1973). The linear logistic test model as an instrument in educational research. Acta Psychologica, 37, 359–374. CrossRefGoogle Scholar
  10. Fischer, G.H. (1976). Some probabilistic models for measuring change. In D.N.M. de Gruijter, & L.J.T. van der Kamp (Eds.), Advances in psychological and educational measurement (pp. 97–110). New York: Wiley. Google Scholar
  11. Fischer, G.H. (1995). Some neglected problems in IRT. Psychometrika, 60, 459–487. CrossRefGoogle Scholar
  12. Fischer, G.H. (2001). Gain scores revisited under an IRT perspective. In A. Boomsma, M.A.J. Van Duijn, & T.A.B. Snijders (Eds.), Essays on item response theory (pp. 43–68). New York: Springer. Google Scholar
  13. Gilula, Z., & Haberman, S.J. (2001). Analysis of categorical response profiles by informative summaries. Sociological Methodology, 31, 193–211. CrossRefGoogle Scholar
  14. Glück, J., & Spiel, C. (1997). Item response models for repeated measures designs: application and limitations of four different approaches. Methods of Psychological Research Online, 2(1), 1–18. Retrieved March 12, 2009, from http://www.dgps.de/fachgruppen/methoden/mpr-online/issue2/art6/article.html. Google Scholar
  15. Hsieh, C., Xu, X., & von Davier, M. (2009). Variance estimation for NAEP data using a resampling-based approach: an application of cognitive diagnostic models. In M. von Davier, & D. Hastedt (Eds.), IERI monograph series: Vol. 2. Issues and methodologies in large scale assessments (pp. 161–174). Hamburg/Princeton: IEA-ETS Research Institute. Google Scholar
  16. Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading: Addison-Wesley. Google Scholar
  17. Masters, G.N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174. CrossRefGoogle Scholar
  18. McLachlan, G., & Peel, D. (2000). Finite mixture models. New York: Wiley. CrossRefGoogle Scholar
  19. Meiser, T., Hein-Eggers, M., Rompe, P., & Rudinger, G. (1995). Analyzing homogeneity and heterogeneity of change using Rasch and latent class models: a comparative and integrative approach. Applied Psychological Measurement, 19(4), 377–391. CrossRefGoogle Scholar
  20. Mislevy, R.J. (1991). Randomization-based inference about latent variables from complex samples. Psychometrika, 56(2), 177–196. CrossRefGoogle Scholar
  21. Muraki, E. (1992). A generalized partial credit model: application of an EM algorithm. Applied Psychological Measurement, 16(2), 159–177. CrossRefGoogle Scholar
  22. Organisation for Economic Co-operation and Development (2003). The PISA 2003 assessment framework: mathematics, reading, science and problem solving knowledge and skills. Paris: Author. Google Scholar
  23. Organisation for Economic Co-operation and Development (2004). Learning for tomorrow’s world: first results from PISA 2003. Paris: Author. Google Scholar
  24. Prenzel, M., Carstensen, C.H., Schöps, K., & Maurischat, C. (2006). Die Anlage des Längsschnitts bei PISA 2003 [The design of the longitudinal PISA assessment]. In M. Prenzel, J. Baumert, W. Blum, R. Lehmann, D. Leutner, M. Neubrand et al. (Eds.), PISA 2003: Untersuchungen zur Kompetenzentwicklung im Verlauf eines Schuljahres [Studies on the development of competencies over the course of a school year] (pp. 29–63). Münster: Waxmann. Google Scholar
  25. Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press. Google Scholar
  26. Rijmen, F. (2009). Efficient full information maximum likelihood estimation for multidimensional IRT models (ETS Research Report No. RR-09-03). Princeton, NJ: ETS. Google Scholar
  27. Rijmen, F., de Boeck, P., & Maas, H. (2005). An IRT model with a parameter-driven process for change. Psychometrika, 70, 651–669. CrossRefGoogle Scholar
  28. Rost, J. (1990). Rasch models in latent classes: an integration of two approaches to item analysis. Applied Psychological Measurement, 14, 271–282. CrossRefGoogle Scholar
  29. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464. CrossRefGoogle Scholar
  30. Stanovich, K.E. (1986). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Reading Research Quarterly, 21(4), 360–407. CrossRefGoogle Scholar
  31. Vermunt, J.K. (2003). Multilevel latent class models. Sociological Methodology, 33, 213–239. CrossRefGoogle Scholar
  32. von Davier, M. (2005). A general diagnostic model applied to language testing data (ETS Research Report No. RR-05-16). Princeton, NJ: ETS. Google Scholar
  33. von Davier, M. (2007a). Mixture general diagnostic models (ETS Research Report No. RR-07-32). ETS: Princeton, NJ. Google Scholar
  34. von Davier, M. (2007b). Hierarchical mixtures of diagnostic models (ETS Research Report No. RR-07-19). ETS: Princeton, NJ. Google Scholar
  35. von Davier, M. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287–307. CrossRefGoogle Scholar
  36. von Davier, M. (2010). Hierarchical mixtures of diagnostic models. Psychological Test and Assessment Modeling, 52(1), 8–28. Retrieved from http://www.psychologie-aktuell.com/fileadmin/download/ptam/1-2010/02_vonDavier.pdf. Google Scholar
  37. von Davier, M., & Rost, J. (1995). Polytomous mixed Rasch models. In G.H. Fischer, & I.W. Molenaar (Eds.), Rasch models: foundations, recent developments, and applications (pp. 371–379). New York: Springer. Google Scholar
  38. von Davier, M., & Rost, J. (2006). Mixture distribution item response models. In C.R. Rao, & S. Sinharay (Eds.), Handbook of statistics: Vol. 26. Psychometrics (pp. 643–661). Amsterdam: Elsevier. Google Scholar
  39. von Davier, M., & Sinharay, S. (2007). An importance sampling EM algorithm for latent regression models. Journal of Educational and Behavioral Statistics, 32(3), 233–251. CrossRefGoogle Scholar
  40. von Davier, M., & Sinharay, S. (2010). Stochastic approximation for latent regression item response models. Journal of Educational and Behavioral Statistics, 35(2), 174–193. CrossRefGoogle Scholar
  41. von Davier, M., & von Davier, A. (2007). A unified approach to IRT scale linkage and scale transformations. Methodology, 3(3), 115–124. Google Scholar
  42. von Davier, M., & Yamamoto, K. (2004). A class of models for cognitive diagnosis. Paper presented at the 4th Spearman invitational conference, October, Philadelphia, PA. Google Scholar
  43. von Davier, M., & Yamamoto, K. (2007). Mixture distribution Rasch models and hybrid Rasch models. In M. von Davier, & C.H. Carstensen (Eds.), Multivariate and mixture distribution rasch models (pp. 99–115). New York: Springer. CrossRefGoogle Scholar
  44. Walberg, H.J., & Tsai, S.-L. (1983). Matthew effects in education. American Educational Research Journal, 20, 359–373. Google Scholar
  45. Wilson, M. (1989). Saltus: a psychometric model for discontinuity in cognitive development. Psychological Bulletin, 105, 276–289. CrossRefGoogle Scholar
  46. Wilson, M., & Draney, K. (1997). Partial credit in a developmental context: the case for adopting a mixture model approach. In M. Wilson, G. Engelhard, Jr., & K. Draney (Eds.), Objective measurement: theory into practice (Vol. 4, pp. 333–350). Greenwich: Ablex. Google Scholar
  47. Xu, X., & von Davier, M. (2006). Cognitive diagnosis for NAEP proficiency data (ETS Research Report No. RR-06-08). Princeton, NJ: ETS. Google Scholar
  48. Xu, X., & von Davier, M. (2008). Comparing multiple-group multinomial loglinear models for multidimensional skill distributions in the general diagnostic model (ETS Research Report No. RR-08-35). Princeton, NJ: ETS. Google Scholar

Copyright information

© The Psychometric Society 2011

Authors and Affiliations

  • Matthias von Davier
    • 1
  • Xueli Xu
    • 1
  • Claus H. Carstensen
    • 2
  1. 1.ETSPrincetonUSA
  2. 2.Bamberg UniversityBambergGermany

Personalised recommendations