Skip to main content

Advertisement

SpringerLink
Log in

A multidimensional IRT approach for dimensionality assessment of standardised students’ tests in mathematics

  • Published:
Quality & Quantity Aims and scope Submit manuscript

Abstract

Mathematics proficiency involves several content domains and processes at different levels. This essentially means that mathematics ability is a complex latent variable. In standardised testing, the complex, and unobserved, latent constructs underlying a test are traditionally appraised by expert panels through subjective measures. In the present research, we deal with the issue of dimensionality of the latent structure behind a test measuring the mathematics ability of Italian students from a statistical and objective point of view, within an IRT framework. The data refer to a national standardised test developed and collected by the Italian National Institute for the Evaluation of the Education System (INVALSI), and administered to lower secondary school students (grade 8). The model we apply is based on a class of multidimensional latent class IRT models, which allows us to ascertain the test dimensionality based on an explorative approach, and by concurrently accounting for non-constant item discrimination and a discrete latent variable formulation. Our results show that the latent abilities underlying the INVALSI test mirror the assessment objectives defined at the national level for the mathematics curriculum. We recommend the use of the proposed extended IRT models in the practice of test construction, primarily—but not exclusively—in the educational field, to support the meaningfulness of the inferences made from test scores about students’ abilities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  • Bacci, S., Bartolucci, F., Gnaldi, M.: A class of multidimensional latent class IRT models for ordinal polytomous item responses. Commun. Stat Theory Methods 43, 787–800 (2014)

    Article  Google Scholar 

  • Bartolini Bussi, M.G., Boni, M., Ferri, F., Garuti, R.: Early approach to theoretical thinking: gears in primary school. Educ. Stud. Math. 39, 67–87 (1999)

    Article  Google Scholar 

  • Bartolucci, F.: A class of multidimensional IRT models for testing unidimensionality and clustering items. Psychometrika 72, 141–157 (2007)

    Article  Google Scholar 

  • Bartolucci, F., Bacci, S., Gnaldi, M.: MultiLCIRT: an R package for multidimensional latent class item response models. Comput. Stat. Data Anal. 71, 971–985 (2014)

    Article  Google Scholar 

  • Bartolucci, F., Bacci, S., Gnaldi, M.: Statistical Analysis of Questionnaires: A Unified Approach Based on R and Stata. Chapman & Hall/CRCHall, New York (2015)

    Book  Google Scholar 

  • Birnbaum, A.: Some latent trait models and their use in inferring an examinee’s ability. In: Lord, F.M., Novick, M.R. (eds.) Statistical Theories of Mental Test Scores, pp. 395–479. Addison-Wesley, Reading, MA (1968)

    Google Scholar 

  • Briggs, D., Wilson, M.: An introduction to multidimensional measurement using Rasch models. J. Appl. Meas. 4, 87–100 (2003)

    Google Scholar 

  • Cai L, Yang JS, Hansen M.: Generalized full-information item bifactor analysis. Psychol. Methods 16, 221–248 (2011)

    Article  Google Scholar 

  • Camilli, G.: A conceptual analysis of differential item functioning in terms of a multidimensional item response model. Appl. Psychol. Meas. 16, 129–147 (1992)

    Article  Google Scholar 

  • Cizek, G., Bunch, M., Koons, H.: Setting performance standards: contemporary methods. Educ. Meas. 23(4), 31–50 (2004)

    Article  Google Scholar 

  • Douek, N.: Some remarks about argumentation and proof. In: Boero, P. (ed.) Theorems in School: From History, Epistemology and Cognition to Classroom Practice. Sense Publishers, Rotterdam (2006)

    Google Scholar 

  • Embretson, S.E.: A multidimensional latent trait model for measuring learning and change. Psychometrika 56, 495–515 (1991)

    Article  Google Scholar 

  • Formann, A.K.: Linear logistic latent class analysis and the Rasch model. In: Fischer, G., Molenaar, I. (eds.) Rasch Models: Foundations, Recent Developments, and Applications, pp. 239–255. Springer, New York (1995)

    Chapter  Google Scholar 

  • Glas, C.A.W., Verhelst, N.D.: Testing the rasch model. In: Fischer, G.H., Molenaar, I. (eds.) Rasch Models. Their Foundations, Recent Developments and Applications, pp. 69–95. Springer, New York (1995)

    Google Scholar 

  • Gnaldi, M., Bartolucci, F., Bacci, S.: A multilevel finite mixture item response model to cluster examinees and schools. Adv. Data Anal. Classif. (2015). doi:10.1007/s11634-014-0196-0

  • Golay, P., Lecerf, T.: On higher order structure and confirmatory factor analysis of the French Wechsler Adult Intelligence Scale (WAIS-III). Psychol. Assess. 23, 143–152 (2011)

    Article  Google Scholar 

  • Holzinger, K., Swineford, S.: The bi-factor method. Psychometrika 47, 41–54 (1937)

    Article  Google Scholar 

  • INVALSI: Quadro di riferimento per il primo ciclo di istruzione. Technical report, INVALSI (2012a)

  • INVALSI: Quadro di riferimento per il secondo ciclo di istruzione. Technical report INVALSI (2012b)

  • Jennrich, R., Bentler, P.: Exploratory bi-factor analysis. Psychometrika 76, 537–549 (2011)

    Article  Google Scholar 

  • Jennrich, R., Bentler, P.: Exploratory bi-factor analysis: the Oblique case. Psychometrika 77, 442–454 (2012)

    Article  Google Scholar 

  • Kane, M.: Content-related validity evidence in test development. In: Downing, S.M., Haladyna, T.M. (eds.) Handbook of Test Development. Lawrence Erlbaum Associates, Mahwah, New Jersey (2006)

    Google Scholar 

  • Lazarsfeld, P.F., Henry, N.W.: Latent Structure Analysis. Houghton Mifflin, Boston (1968)

    Google Scholar 

  • Lindsay, B., Clogg, C., Greco, J.: Semiparametric estimation in the rasch model and related exponential response models, including a simple latent class model for item analysis. J. Am. Stat. Assoc. 86, 96–107 (1991)

    Article  Google Scholar 

  • Loomis, S., Bourque, M.: From tradition to innovation: Standard setting on the national assessment of educational progress. In: Cizek, G.J. (ed.) Setting Performance Standards: Concepts Methods and Perspectives. Lawrence Erlbaum Associates, Mahwah, NJ (2001)

    Google Scholar 

  • Luecht, R.M., Miller, R.: Unidimensional calibrations and interpretations of composite traits for multidimensional tests. Appl. Psychol. Meas. 16, 279–293 (1992)

    Article  Google Scholar 

  • Martin-Löf, P.: Statistiska Modeller. Institütet för Försäkringsmatemetik och Matematisk Statistisk vid Stockholms Universitet, Stockholm (1973)

    Google Scholar 

  • Matteucci M, Mignani S (2015) Multidimensional irt models to analyze learning outcomes of italian students at the end of lower secondary school. In: Millsap R, Bolt D, van der Ark L, Wang W (eds) Quantitative Psychology Research, Springer Proceedings in Mathematics & Statistics, Springer International Publishing Switzerland, vol 89, pp. 91–111

  • Messick, S.: Validity. In: Linn, R.L. (ed.) Educational Measurement. American Council on Education and Macmillan, New York (1989)

    Google Scholar 

  • Mokken, R.: A Theory and Procedure of Scale Analysis. De Gruyter, Berlin, Germany (1971)

    Book  Google Scholar 

  • Rasch G (1961) On general laws and the meaning of measurement in psychology. In: Proceedings of the IV Berkeley Symposium on Mathematical Statistics and Probability, University of California Press, pp. 321–333

  • Raykov, T., Marcoulides, G.A.: Introduction to Psychometric Theory. Routledge, Taylor & Francis Group, New York (2011)

    Google Scholar 

  • Reckase, M.: Multidimensional Item Response Theory. Springer, NewYork (2009)

    Book  Google Scholar 

  • Schoenfeld, A.: Learning to think mathematically: problem solving, metacognition, and sense making in mathematics. In: Grows, D. (ed.) Handbook for Research on Mathematics Teaching and Learning. Macmillan, New York (1992)

    Google Scholar 

  • Schwarz, G.: Estimating the dimension of a model. Ann. Stat. 6, 461–464 (1978)

    Article  Google Scholar 

  • Sijtsma, K., Molenaar, I.: Introduction to Nonparametric Item Response Theory. Sage, Thousand Oaks (2002)

    Book  Google Scholar 

  • Stout, W.: A non parametric approach for assessing latent trait unidimensionality. Psychometrika 52(4), 589–617 (1987)

    Article  Google Scholar 

  • Tout, D., Spithill, J.: The challenges and complexities of writing items to test mathematical literacy. In: Turner, R., Stacey, K. (eds.) Assessing Mathematical Literacy, The PISA Experience. Springer, New York (2014)

    Google Scholar 

  • Vermunt, J.: The use of restricted latent class models for defining and testing nonparametric and parametric item response theory models. Appl. Psychol. Meas. 25, 283–294 (2001)

    Article  Google Scholar 

  • Webb, N.L.: Identifying content for student achievement tests. In: Downing, S.M., Haladyna, T.M. (eds.) Handbook of Test Development. Lawrence Erlbaum Associates, Mahwah, New Jersey (2006)

    Google Scholar 

  • Wirth, R., Edwards, M.: Item factor analysis: current approaches and future directions. Psychol. Methods 12(1), 58–79 (2007)

    Article  Google Scholar 

  • Zhang, J., Stout, W.: Conditional covariance structure of generalized compensatory multidimensional item. Psychometrika 64(2), 129–152 (1999a)

    Article  Google Scholar 

  • Zhang, J., Stout, W.: The theoretical detect index of dimensionality and its application to approximate simple structure. Psychometrika 64(2), 213–249 (1999b)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Political Sciences, University of Perugia, Via A. Pascoli 20, 06123, Perugia, Italy

    Michela Gnaldi

Authors
  1. Michela Gnaldi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michela Gnaldi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gnaldi, M. A multidimensional IRT approach for dimensionality assessment of standardised students’ tests in mathematics. Qual Quant 51, 1167–1182 (2017). https://doi.org/10.1007/s11135-016-0323-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11135-016-0323-4

Keywords

Search

Navigation