Advertisement

Behavior Research Methods

, Volume 50, Issue 1, pp 39–56 | Cite as

Testing the actual equivalence of automatically generated items

  • Debora de Chiusole
  • Luca Stefanutti
  • Pasquale Anselmi
  • Egidio Robusto
Article
  • 131 Downloads

Abstract

If the automatic item generation is used for generating test items, the question of how the equivalence among different instances may be tested is fundamental to assure an accurate assessment. In the present research, the question was dealt by using the knowledge space theory framework. Two different ways of considering the equivalence among instances are proposed: The former is at a deterministic level and it requires that all the instances of an item template must belong to exactly the same knowledge states; the latter adds a probabilistic level to the deterministic one. The former type of equivalence can be modeled by using the BLIM with a knowledge structure assuming equally informative instances; the latter can be modeled by a constrained BLIM. This model assumes equality constraints among the error parameters of the equivalent instances. An approach is proposed for testing the equivalence among instances, which is based on a series of model comparisons. A simulation study and an empirical application show the viability of the approach.

Keywords

Knowledge space theory Basic local independence model Knowledge assessment Automatic item generation Item template 

Notes

Acknowledgements

The research developed in this article was carried out under the research project CPDR152105 “Learning how students learn. Mathematical modeling of learning processes in intelligent tutoring system navigation”, funded by the University of Padua, Italy.

References

  1. Albert, D., & Lukas, J. (1999). Knowledge spaces: Theories, empirical research, and applications. Mahwah, New Jersey, London: Lawrence Erlbaum Associates.Google Scholar
  2. Anselmi, P., Robusto, E., Stefanutti, L., & de Chiusole, D. (2016). An upgrading procedure for adaptive assessment of knowledge. Psychometrika, 81(2), 461–482.CrossRefPubMedGoogle Scholar
  3. Anselmi, P., Stefanutti, L., de Chiusole, D., & Robusto, E. (2017). The assessment of knowledge and learning in competence spaces: The gain–loss model for dependent skills. British Journal of Mathematical and Statistical Psychology, 70(3), 457–479.CrossRefPubMedGoogle Scholar
  4. Arendasy, M. E., & Sommer, M. (2012). Using automatic item generation to meet the increasing item demands of high-stakes educational and occupational assessment. Learning and individual differences, 22(1), 112–117.CrossRefGoogle Scholar
  5. Bolt, D. (2007). The present and future of IRT-based cognitive diagnostic models (ICDMs) and related methods. Journal of Educational Measurement, 44(4), 377–383.CrossRefGoogle Scholar
  6. Bormuth, J. (1969). On a theory of achievement test items. Chicago: University of Chicago Press.Google Scholar
  7. Cosyn, E., Doble, C., Falmagne, J. C., Lenoble, A., Thiéry, N., & Uzun, H. (2013) Assessing mathematical knowledge in a learning space. Knowledge Spaces (pp. 27–50). Berlin: Springer.Google Scholar
  8. Cui, Y., Gierl, M. J., & Chang, H. H. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49(1), 19–38.CrossRefGoogle Scholar
  9. de Chiusole, D., Anselmi, P., Stefanutti, L., & Robusto, E. (2013). The gain–loss model: bias and variance of the parameter estimates. Electronic Notes in Discrete Mathematics, 42, 33–40.CrossRefGoogle Scholar
  10. de Chiusole, D., & Stefanutti, L. (2013). Modeling skill dependence in probabilistic competence structures. Electronic Notes in Discrete Mathematics, 42, 41–48.CrossRefGoogle Scholar
  11. de Chiusole, D., Stefanutti, L., Anselmi, P., & Robusto, E. (2013). Assessing parameter invariance in the BLIM: Bipartition models. Psychometrika, 78(4), 710–724.  https://doi.org/10.1007/s11336-013-9325-5 CrossRefPubMedGoogle Scholar
  12. de Chiusole, D., Stefanutti, L., Anselmi, P., & Robusto, E. (2015a). Modeling missing data in knowledge space theory. Psychological Methods, 20(4), 506–522.Google Scholar
  13. de Chiusole, D., Stefanutti, L., Anselmi, P., & Robusto, E. (2015b). Naïve tests of basic local independence model’s invariance. The Spanish Journal of Psychology, 18, E26.  https://doi.org/10.1017/sjp.2015.24
  14. DiBello, L. V., & Stout, W. (2007). Guest editors’ introduction and overview: IRT-based cognitive diagnostic models and related methods. Journal of Educational Measurement, 44(4), 285–291.CrossRefGoogle Scholar
  15. Doignon, J. P. (1994). Knowledge spaces and skill assignments. In G. Fischer, & D. Laming (Eds.) Contributions to Mathematical Psychology Psychometrics and Methodology (pp. 111–121). New York: Springer.Google Scholar
  16. Doignon, J. P., & Falmagne, J. C. (1985). Spaces for the assessment of knowledge. International Journal of Man-Machine Studies, 23, 175–196.CrossRefGoogle Scholar
  17. Doignon, J. P., & Falmagne, J. C. (1999) Knowledge spaces. New York: Springer.CrossRefGoogle Scholar
  18. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7, 1–26.Google Scholar
  19. Embretson, S. (1999). Generating items during testing: Psychometric issues and models. Psychometrika, 64(4), 407–433.CrossRefGoogle Scholar
  20. Embretson, S. (2005). Measuring human intelligence with artificial intelligence. In R. J. Sternberg, & J. E. Pretz (Eds.) Cognition and intelligence. New York: Cambridge University Press.Google Scholar
  21. Embretson, S., & Yang, X. (2006). Automatic item generation and cognitive psychology. Handbook of statistics: Vol. 26 Psychometrics (pp. 747–768). North Holland: Elsevier.Google Scholar
  22. Falmagne, J. C., & Doignon, J. P. (1988). A class of stochastic procedures for the assessment of knowledge. British Journal of Mathematical and Statistical Psychology, 41, 1–23.CrossRefGoogle Scholar
  23. Falmagne, J. C., & Doignon, J. P. (1988). A Markovian procedure for assessing the state of a system. Journal of Mathematical Psychology, 32, 232–258.CrossRefGoogle Scholar
  24. Falmagne, J. C., & Doignon, J. P. (2011). Learning spaces. New York: Springer.CrossRefGoogle Scholar
  25. Gierl, M. J., & Haladyna, T. M. (2013). Automatic item generation: Theory and practice. Routledge: New York.Google Scholar
  26. Glas, C. A., & van der Linden, W. J. (2003). Computerized adaptive testing with item cloning. Applied Psychological Measurement, 27(4), 247–261.CrossRefGoogle Scholar
  27. Gulliksen, H. (2013). Theory of mental tests. Abingdon: Routledge.Google Scholar
  28. Heller, J. (2017). Identifiability in probabilistic knowledge structures. Journal of Mathematical Psychology, 77, 46–57.CrossRefGoogle Scholar
  29. Heller, J., Stefanutti, L., Anselmi, P., & Robusto, E. (2015). On the link between cognitive diagnostic models and knowledge space theory. Psychometrika, 80(4), 995–1019.CrossRefPubMedGoogle Scholar
  30. Heller, J., Stefanutti, L., Anselmi, P., & Robusto, E. (2016). Erratum to: on the link between cognitive diagnostic models and knowledge space theory. Psychometrika, 81(1), 250–251.CrossRefPubMedGoogle Scholar
  31. Heller, J., & Wickelmaier, F. (2013). Minimum discrepancy estimation in probabilistic knowledge structures. Electronic Notes in Discrete Mathematics, 42(4), 49–56.CrossRefGoogle Scholar
  32. Irvine, S. H., & Kyllonen, P. C. (2013) Item generation for test development. Abingdon: Routledge.Google Scholar
  33. Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272.CrossRefGoogle Scholar
  34. Korossy, K. (1999). Modeling knowledge as competence and performance. In D. Albert, & J. Lukas (Eds.) Knowledge spaces: Theories, empirical research, applications (pp. 103–132). Mahwah: Lawrence Erlbaum Associates.Google Scholar
  35. Robusto, E., Anselmi, P., & Stefanutti, L. (2010). The gain–loss model: A probabilistic skill multimap model for assessing learning processes. Journal of Educational Measurement, 47(3), 373–394.CrossRefGoogle Scholar
  36. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581–592.CrossRefGoogle Scholar
  37. Rudner, L. M. (2010). Implementing the graduate management admission test computerized adaptive test. Elements of adaptive testing (pp. 151–165). New York: Springer.Google Scholar
  38. Schrepp, M. (2005). About the connection between knowledge structures and latent class models. Methodology, 1, 93–103.CrossRefGoogle Scholar
  39. Singley, M. K., & Bennett, R. E. (2002). Item generation and beyond: Applications of schema theory to mathematics assessment. Item generation for test development (pp. 361–384). Mahwah: Lawrence Erlbaum.Google Scholar
  40. Sinharay, S., & Johnson, M. S. (2013). Statistical modeling of automatically generated items. Automatic item generation: Theory and practice (pp. 183–195). Routledge: New York.Google Scholar
  41. Spoto, A., Stefanutti, L., & Vidotto, G. (2012). On the unidentifiability of a certain class of skill multi map based probabilistic knowledge structures. Journal of Mathematical Psychology, 56(4), 248–255.CrossRefGoogle Scholar
  42. Spoto, A., Stefanutti, L., & Vidotto, G. (2013). Considerations about the identification of forward-and backward-graded knowledge structures. Journal of Mathematical Psychology, 57(5), 249–254.CrossRefGoogle Scholar
  43. Stefanutti, L., Heller, J., Anselmi, P., & Robusto, E. (2012). Assessing local identifiability of probabilistic knowledge structures. Behavior Research Methods, 44(4), 1197–1211.CrossRefPubMedGoogle Scholar
  44. Stefanutti, L., & Robusto, E. (2009). Recovering a probabilistic knowledge structure by constraining its parameter space. Psychometrika, 74, 83–96.CrossRefGoogle Scholar
  45. Tatsuoka, K. (1990). Toward an integration of item-response theory and cognitive error diagnosis. In Monitoring skills and knowledge acquisition (pp. 453–488). Lawrence Erlbaum Associates: Hillsdale.Google Scholar
  46. Ünlü, A., Schrepp, M., Heller, J., Hockemeyer, C., Wesiak, G., & Albert, D. (2013). Recent developments in performance-based knowledge space theory. Knowledge Spaces (pp. 147–192). Berlin: Springer.Google Scholar

Copyright information

© Psychonomic Society, Inc. 2018

Authors and Affiliations

  • Debora de Chiusole
    • 1
  • Luca Stefanutti
    • 1
  • Pasquale Anselmi
    • 1
  • Egidio Robusto
    • 1
  1. 1.University of Padua (Italy)PaduaItaly

Personalised recommendations