Mapping Problems to Skills Combining Expert Opinion and Student Data

  • Juraj NižnanEmail author
  • Radek Pelánek
  • Jiří Řihák
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8934)


Construction of a mapping between educational content and skills is an important part of development of adaptive educational systems. This task is difficult, requires a domain expert, and any mistakes in the mapping may hinder the potential of an educational system. In this work we study techniques for improving a problem-skill mapping constructed by a domain expert using student data, particularly problem solving times. We describe and compare different techniques for the task – a multidimensional model of problem solving times and supervised classification techniques. In the evaluation we focus on surveying situations where the combination of expert opinion with student data is most useful.


Logistic Regression Expert Opinion Recommender System Item Response Theory Spectral Cluster 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Jarušek, P., Pelánek, R.: Analysis of a simple model of problem solving times. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 379–388. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  2. 2.
    Jarušek, P., Pelánek, R.: A web-based problem solving tool for introductory computer science. In: Proc. of Innovation and technology in computer science education, pp. 371–371. ACM (2012)Google Scholar
  3. 3.
    Barnes, T.: The q-matrix method: mining student response data for knowledge. In: American Association for Artificial Intelligence 2005 Educational Data Mining Workshop (2005)Google Scholar
  4. 4.
    Desmarais, M.C., Beheshti, B., Naceur, R.: Item to skills mapping: deriving a conjunctive q-matrix from data. In: Cerri, S.A., Clancey, W.J., Papadourakis, G., Panourgia, K. (eds.) ITS 2012. LNCS, vol. 7315, pp. 454–463. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  5. 5.
    Tatsuoka, K.: Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement 20(4), 345–354 (1983)CrossRefGoogle Scholar
  6. 6.
    Barnes, T.: Novel derivation and application of skill matrices: The q-matrix method. In: Handbook on Educational Data Mining (2010)Google Scholar
  7. 7.
    Boroš, P., Nižnan, J., Pelánek, R., Řihák, J.: Automatic detection of concepts from problem solving times. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS, vol. 7926, pp. 595–598. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  8. 8.
    De La Torre, J.: An empirically based method of q-matrix validation for the dina model: Development and applications. Journal of Educational Measurement 45(4), 343–362 (2008)CrossRefGoogle Scholar
  9. 9.
    DeCarlo, L.T.: Recognizing uncertainty in the q-matrix via a bayesian extension of the dina model. Applied Psychological Measurement 36(6), 447–468 (2012)CrossRefGoogle Scholar
  10. 10.
    Desmarais, M.C., Naceur, R.: A matrix factorization method for mapping items to skills and for enhancing expert-based Q-matrices. In: Lane, H.C., Yacef, K., Mostow, J., Pavlik, P. (eds.) AIED 2013. LNCS, vol. 7926, pp. 441–450. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  11. 11.
    Rupp, A., Templin, J.: The effects of q-matrix misspecification on parameter estimates and classification accuracy in the dina model. Educational and Psychological Measurement 68(1), 78–96 (2008)CrossRefMathSciNetGoogle Scholar
  12. 12.
    Desmarais, M.C., Beheshti, B., Xu, P.: The refinement of a q-matrix: assessing methods to validate tasks to skills mapping. In: Proceedings of the 7th International Conference on Educational Data Mining, pp. 308–311 (2014)Google Scholar
  13. 13.
    Kantor, P., Ricci, F., Rokach, L., Shapira, B.: Recommender systems handbook. Springer (2010)Google Scholar
  14. 14.
    Waters, A., Lan, A., Studer, C.: Sparse probit factor analysis for learning analytics. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 8776–8780 (2013)Google Scholar
  15. 15.
    Cen, H., Koedinger, K.R., Junker, B.: Learning factors analysis – a general method for cognitive model evaluation and improvement. In: Ikeda, M., Ashley, K.D., Chan, T.-W. (eds.) ITS 2006. LNCS, vol. 4053, pp. 164–175. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  16. 16.
    Lindsey, R.V., Khajah, M., Mozer, M.C.: Automatic discovery of cognitive skills to improve the prediction of student learning (2014), submitted for publicationGoogle Scholar
  17. 17.
    Bootkrajang, J., Kabán, A.: Label-noise robust logistic regression and its applications. In: Flach, P.A., De Bie, T., Cristianini, N. (eds.) ECML PKDD 2012, Part I. LNCS, vol. 7523, pp. 143–158. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  18. 18.
    Natarajan, N., Dhillon, I., Ravikumar, P., Tewari, A.: Learning with noisy labels. In: Advances in Neural Information Processing Systems, pp. 1196–1204 (2013)Google Scholar
  19. 19.
    Nižnan, J., Pelánek, R., Řihák, J.: Using problem solving times and expert opinion to detect skills. In: Proceedings of the 7th International Conference on Educational Data Mining, pp. 433–434 (2014)Google Scholar
  20. 20.
    Van der Linden, W.: A lognormal model for response times on test items. Journal of Educational and Behavioral Statistics 31(2), 181 (2006)CrossRefMathSciNetGoogle Scholar
  21. 21.
    Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8), 30–37 (2009)CrossRefGoogle Scholar
  22. 22.
    Fan, R.E., Chang, K.W., Hsieh, C.J., Wang, X.R., Lin, C.J.: Liblinear: A library for large linear classification. The Journal of Machine Learning Research 9, 1871–1874 (2008)zbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  1. 1.Masaryk UniversityBrnoCzech Republic

Personalised recommendations