Evaluation of Expert-Based Q-Matrices Predictive Quality in Matrix Factorization Models

  • Guillaume DurandEmail author
  • Nabil Belacel
  • Cyril Goutte
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9307)


Matrix factorization techniques are widely used to build collaborative filtering recommender systems. These recommenders aim at discovering latent variables or attributes that are supposed to explain and ultimately predict the interest of users. In cognitive modeling, skills and competencies are considered as key latent attributes to understand and assess student learning. For this purpose, Tatsuoka introduced the concept of Q-matrix to represent the mapping between skills and test items. In this paper we evaluate how predictive expert-created Q-matrices can be when used as a decomposition factor in a matrix factorization recommender. To this end, we developed an evaluation method using cross validation and the weighted least squares algorithm that measures the predictive accuracy of Q-matrices. Results show that expert-made Q-matrices can be reasonably accurate at predicting users success in specific circumstances that are discussed at the end of this paper.


Cognitive models Matrix factorization Recommender systems Competency-based learning 



This work is part of the National Research Council Canada program Learning and Performance Support Systems (LPSS), which addresses training, development and performance support in all industry sectors, including education, oil and gas, policing, military and medical devices.


  1. 1.
    Barnes, T.: The Q-matrix method: Mining student response data for knowledge. In: AAAI Educational Data Mining workshop. p. 39 (2005)Google Scholar
  2. 2.
    Beheshti, B., Desmarais, M.C.: Predictive performance of prevailing approaches to skills assessment techniques: Insights from real vs. synthetic data sets (2014)Google Scholar
  3. 3.
    Birenbaum, M., Kelly, A.E., Tatsuoka, K.K.: Diagnosing Knowledge States in Algebra Using the Rule Space Model. ETS research report, Educational Testing Service, Educational Testing Service Princeton, NJ (1992)Google Scholar
  4. 4.
    DeCarlo, L.T.: On the analysis of fraction subtraction data: the DINA model, classification, latent class sizes, and the Q-matrix. Appl. Psychol. Meas. 35(1), 8–26 (2011)CrossRefGoogle Scholar
  5. 5.
    Desmarais, M.C., Beheshti, B., Naceur, R.: Item to skills mapping: deriving a conjunctive Q-matrix from data. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 454–463. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  6. 6.
    Desmarais, M.C., Naceur, R.: A matrix factorization method for mapping items to skills and for enhancing expert-based Q-Matrices. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS, vol. 7926, pp. 441–450. Springer, Heidelberg (2013) CrossRefGoogle Scholar
  7. 7.
    Gierl, M.J., Leighton, J.P., Hunka, S.M.: Exploring the logic of Tatsuoka’s rule-space model for test development and analysis. Educ. Meas. Issues Pract. 19(3), 34–44 (2000). CrossRefGoogle Scholar
  8. 8.
    Junker, B.W., Sijtsma, K.: Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Appl. Psychol. Meas. 25(3), 258–272 (2001). MathSciNetCrossRefGoogle Scholar
  9. 9.
    Koedinger, K.R., Baker, R.S.J.D., Cunningham, K., Skogsholm, A., Leber, B., Stamper, J., Ventura, S., Pechenizkiy, M., Baker, R.S.J.D.: A data repository for the EDM community: The PSLC DataShop. In: Romero, C. (ed.) Handbook of Educational Data Mining, pp. 43–56. CRC Press, Boca Raton (2010)CrossRefGoogle Scholar
  10. 10.
    Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8), 30–37 (2009). CrossRefGoogle Scholar
  11. 11.
    Lan, A., Studer, C., Baraniuk, R.: Quantized matrix completion for personalized learning. In: 7th International Conference on Educational Data Mining, pp. 280–283 (2014)Google Scholar
  12. 12.
    Liu, J., Xu, G., Ying, Z.: Data-driven learning of Q-matrix. Appl. Psychol. Meas. 36(7), 548–564 (2012)CrossRefGoogle Scholar
  13. 13.
    Mislevy, R.J.: Test theory reconceived. J. Educ. Meas. 33(4), 379–416 (1996). CrossRefGoogle Scholar
  14. 14.
    Nelder, J.A., Mead, R.: A simplex method for functional minimization. Comput. J. 7, 308–313 (1965)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Ritter, S., Anderson, J., Koedinger, K., Corbett, A.: Cognitive tutor: applied research in mathematics education. Psychon. Bull. Rev. 14(2), 249–255 (2007). CrossRefGoogle Scholar
  16. 16.
    Rupp, A.A., Templin, J.: The effects of q-matrix misspecification on parameter estimates and classification accuracy in the dina model. Educ. Psychol. Meas. 68(1), 78–96 (2008). MathSciNetCrossRefGoogle Scholar
  17. 17.
    Stamper, J., Ritter, S.: Cog model discovery experiment fall 2011. dataset 605 in datashop (2013). Accessed
  18. 18.
    Stamper, J.C., Koedinger, K.R.: Human-machine student model discovery and improvement using DataShop. In: Biswas, G., Bull, S., Kay, J., Mitrovic, A. (eds.) AIED 2011. LNCS, vol. 6738, pp. 353–360. Springer, Heidelberg (2011) CrossRefGoogle Scholar
  19. 19.
    Su, Y.L., Choi, K.M., Lee, W.C., Choi, T., McAninch, M.: Hierarchical cognitive diagnostic analysis for TIMSS] 2003 mathematics. Technical report 35, Center for Advanced Studies in Measurement and Assessment (CASMA), University of Iowa (2013)Google Scholar
  20. 20.
    Sun, Y., Ye, S., Inoue, S., Sun, Y.: Alternating recursive method for Q-matrix learning. In: 7th International Conference on Educational Data Mining, pp. 14–20 (2014)Google Scholar
  21. 21.
    Tatsuoka, K.K.: Rule space: an approach for dealing with misconceptions based on item response theory. J. Educ. Meas. 20(4), 345–354 (1983)CrossRefGoogle Scholar
  22. 22.
    Tatsuoka, K.K.: Analysis of errors in fraction addition and subtraction problems. Final report, Computer-based Education Research Laboratory, University of Illinois at Urbana-Champaign (1984)Google Scholar
  23. 23.
    Tatsuoka, K.K.: Toward an integration of item-response theory and cognitive error diagnosis. In: Frederiksen, N., et al. (eds.) Diagnostic monitoring of skill and knowledge acquisition, pp. 453–488. Lawrence Erlbaum Associates, Hillsdale, NJ (1990) Google Scholar
  24. 24.
    Tatsuoka, K.K.: Item Construction and Psychometric Models Appropriate for Constructed Responses. ETS research report, Educational Testing Service, Educational Testing Service Princeton, NJ (1991)Google Scholar
  25. 25.
    Tatsuoka, K.K.: Architecture of knowledge structures and cognitive diagnosis: a statistical pattern recognition and classification approach. In: Nichols, P.D., Chipman, S.F., Brennan, R. (eds.) Cognitively Diagnostic Assessment, pp. 327–359. Lawrence Erlbaum Associates, Hillsdale (1995)Google Scholar
  26. 26.
    The MathWorks, I.: Matlab 7, function reference. In: Matlab 7. Natick, Massachusetts (2008)Google Scholar
  27. 27.
    de la Torre, J.: An empirically based method of q-matrix validation for the dina model: development and applications. J. Educ. Meas. 45(4), 343–362 (2008). CrossRefGoogle Scholar
  28. 28.
    de la Torre, J.: DINA model and parameter estimation: a didactic. J. Educ. Behav. Stat. 34(1), 115–130 (2009)CrossRefGoogle Scholar
  29. 29.
    de la Torre, J., Douglas, J.: Model evaluation and multiple strategies in cognitive diagnosis: an analysis of fraction subtraction data. Psychometrika 73(4), 595–624 (2008)MathSciNetCrossRefzbMATHGoogle Scholar
  30. 30.
    de la Torre, J., Douglas, J.A.: Higher-order latent trait models for cognitive diagnosis. Psychometrika 69(3), 333–353 (2004)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.National Research Council CanadaInformation and Communications TechnologiesMonctonCanada
  2. 2.National Research Council CanadaInformation and Communications TechnologiesOttawaCanada

Personalised recommendations