Which Algorithms Suit Which Learning Environments? A Comparative Study of Recommender Systems in TEL

  • Simone KopeinikEmail author
  • Dominik Kowald
  • Elisabeth Lex
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9891)


In recent years, a number of recommendation algorithms have been proposed to help learners find suitable learning resources on-line. Next to user-centered evaluations, offline-datasets have been used to investigate new recommendation algorithms or variations of collaborative filtering approaches. However, a more extensive study comparing a variety of recommendation strategies on multiple TEL datasets is missing. In this work, we contribute with a data-driven study of recommendation strategies in TEL to shed light on their suitability for TEL datasets. To that end, we evaluate six state-of-the-art recommendation algorithms for tag and resource recommendations on six empirical datasets: a dataset from European Schoolnets TravelWell, a dataset from the MACE portal, which features access to meta-data-enriched learning resources from the field of architecture, two datasets from the social bookmarking systems BibSonomy and CiteULike, a MOOC dataset from the KDD challenge 2015, and Aposdle, a small-scale workplace learning dataset. We highlight strengths and shortcomings of the discussed recommendation algorithms and their applicability to the TEL datasets. Our results demonstrate that the performance of the algorithms strongly depends on the properties and characteristics of the particular dataset. However, we also find a strong correlation between the average number of users per resource and the algorithm performance. A tag recommender evaluation experiment reveals that a hybrid combination of a cognitive-inspired and a popularity-based approach consistently performs best on all TEL datasets we utilized in our study.


Offline study Tag recommendation Resource recommendation Recommender systems ACT-R SUSTAIN Technology enhanced learning TEL 



We would like to gratefully acknowledge Katja Niemann who provided us with the MACE and TravelWell datasets, as well as the organizers of KDD Cup 2015 and XuetangX for making the KDD dataset available. This work is funded by the Know-Center, the EU-IP Learning Layers (Grant Agreement: 318209) and the EU-IP AFEL (Grant Agreement: 687916). The Know-Center is funded within the Austrian COMET Program under the auspices of the Austrian Ministry of Transport, Innovation and Technology, the Austrian Ministry of Economics and Labor and by the State of Styria.


  1. 1.
    Drachsler, H., Verbert, K., Santos, O.C., Manouselis, N.: Panorama of recommender systems to support learning. In: Ricci, F., Rokach, L., Shapira, B. (eds.) Recommender Systems Handbook, pp. 421–451. Springer, Heidelberg (2015)CrossRefGoogle Scholar
  2. 2.
    Khribi, M.K., Jemni, M., Nasraoui, O.: Recommendation systems for personalized technology-enhanced learning. In: Kinshuk, Huang, R. (eds.) Ubiquitous Learning Environments and Technologies, pp. 159–180. Springer, Heidelberg (2015)Google Scholar
  3. 3.
    Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., Koper, R.: Recommender systems in technology enhanced learning. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 387–415. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  4. 4.
    Verbert, K., Manouselis, N., Drachsler, H., Duval, E.: Dataset-driven research to support learning and knowledge analytics. Educ. Technol. Soc. 15(3), 133–148 (2012)Google Scholar
  5. 5.
    Verbert, K., Drachsler, H., Manouselis, N., Wolpers, M., Vuorikari, R., Duval, E.: Dataset-driven research for improving recommender systems for learning. In: Proceedings of LAK 2011, pp. 44–53. ACM (2011)Google Scholar
  6. 6.
    Fazeli, S., Loni, B., Drachsler, H., Sloep, P.: Which recommender system can best fit social learning platforms? In: Rensing, C., de Freitas, S., Ley, T., Muñoz-Merino, P.J. (eds.) EC-TEL 2014. LNCS, vol. 8719, pp. 84–97. Springer, Heidelberg (2014)Google Scholar
  7. 7.
    Niemann, K., Wolpers, M.: Usage context-boosted filtering for recommender systems in TEL. In: Hernández-Leo, D., Ley, T., Klamma, R., Harrer, A. (eds.) EC-TEL 2013. LNCS, vol. 8095, pp. 246–259. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  8. 8.
    Manouselis, N., Vuorikari, R., Van Assche, F.: Collaborative recommendation of e-learning resources: an experimental investigation. J. Comput. Assist. Learn. 26(4), 227–242 (2010)CrossRefGoogle Scholar
  9. 9.
    Bateman, S., Brooks, C., Mccalla, G., Brusilovsky, P.: Applying collaborative tagging to e-learning. In: Proceedings WWW 2007 (2007)Google Scholar
  10. 10.
    Kuhn, A., McNally, B., Schmoll, S., Cahill, C., Lo, W.-T., Quintana, C., Delen, I.: How students find, evaluate and utilize peer-collected annotated multimedia data in science inquiry with Zydeco. In: Proceedings of SIGCHI 2012, pp. 3061–3070. ACM (2012)Google Scholar
  11. 11.
    Klašnja-Milićević, A., Ivanović, M., Nanopoulos, A.: Recommender systems in e-learning environments: a survey of the state-of-the-art and possible extensions. Artif. Intell. Rev. 44(4), 571–604 (2015)CrossRefGoogle Scholar
  12. 12.
    Manouselis, N., Drachsler, H., Verbert, K., Duval, E.: Recommender Systems for Learning. Springer, New York (2012)Google Scholar
  13. 13.
    Erdt, M., Fernandez, A., Rensing, C.: Evaluating recommender systems for technology enhanced learning: a quantitative survey. IEEE Trans. Learn. Technol. 8(4), 326–344 (2015)CrossRefGoogle Scholar
  14. 14.
    Lohmann, S., Thalmann, S., Harrer, A., Maier, R.: Learner-generated annotation of learning resources-lessons from experiments on tagging. J. Univ. Comput. Sci. 304, 312 (2007)Google Scholar
  15. 15.
    Diaz-Aviles, E., Fisichella, M., Kawase, R., Nejdl, W., Stewart, A.: Unsupervised auto-tagging for learning object enrichment. In: Kloos, C.D., Gillet, D., Crespo García, R.M., Wild, F., Wolpers, M. (eds.) EC-TEL 2011. LNCS, vol. 6964, pp. 83–96. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  16. 16.
    Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)zbMATHGoogle Scholar
  17. 17.
    Niemann, K.: Automatic tagging of learning objects based on their usage in web portals. In: Conole, G., Klobučar, T., Rensing, C., Konert, J., Lavoué, E. (eds.) Design for Teaching and Learning in a Networked World, vol. 9307, pp. 240–253. Springer, Heidelberg (2015)Google Scholar
  18. 18.
    Kowald, D., Lex, E.: Evaluating tag recommender algorithms in real-world folksonomies: a comparative study. In: Proceedings of RecSys 2015, pp. 265–268. ACM (2015)Google Scholar
  19. 19.
    Seitlinger, P., Kowald, D., Kopeinik, S., Hasani-Mavriqi, I., Ley, T., Lex, E.: Attention please! a hybrid resource recommender mimicking attention-interpretation dynamics. In: Proceedings of International World Wide Web Conferences Steering Committee, WWW 2015, pp. 339–345 (2015)Google Scholar
  20. 20.
    Trattner, C., Kowald, D., Seitlinger, P., Kopeinik, S., Ley, T.: Modeling activation processes in human memory to predict the use of tags in social bookmarking systems. J. Web Sci. 2(1), 1–16 (2016)CrossRefGoogle Scholar
  21. 21.
    Kowald, D., Lacic, E., Trattner, C.: Tagrec: towards a standardized tagrecommender benchmarking framework. In: Proceedings of HT 2014. ACM, New York (2014)Google Scholar
  22. 22.
    Jäschke, R., Marinho, L., Hotho, A., Schmidt-Thieme, L., Stumme, G.: Tag recommendations in Folksonomies. In: Kok, J.N., Koronacki, J., Lopez de Mantaras, R., Matwin, S., Mladenič, D., Skowron, A. (eds.) PKDD 2007. LNCS (LNAI), vol. 4702, pp. 506–514. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  23. 23.
    Schafer, J.B., Frankowski, D., Herlocker, J., Sen, S.: Collaborative filtering recommender systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) Adaptive Web 2007. LNCS, vol. 4321, pp. 291–324. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  24. 24.
    Marinho, L.B., Schmidt-Thieme, L.: Collaborative tag recommendations. In: Preisach, C., Burkhardt, H., Schmidt-Thieme, L., Decker, R. (eds.) Data Analysis, Machine Learning and Applications, pp. 533–540. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  25. 25.
    Gemmell, J., Schimoler, T., Ramezani, M., Christiansen, L., Mobasher, B.: Improving folkrank with item-based collaborative filtering. In: Recommender Systems & the Social Web (2009)Google Scholar
  26. 26.
    Basilico, J., Hofmann, T.: Unifying collaborative and content-based filtering. In: Proceedings of ICML 2004, p. 9. ACM (2004)Google Scholar
  27. 27.
    Friedrich, M., Niemann, K., Scheffel, M., Schmitz, H.-C., Wolpers, M.: Object recommendation based on usage context. Educ. Technol. Soc. 10(3), 106–121 (2007)Google Scholar
  28. 28.
    Niemann, K., Wolpers, M.: Creating usage context-based object similarities to boost recommender systems in technology enhanced learning. IEEE Trans. Learn. Technol. 8(3), 274–285 (2015)CrossRefGoogle Scholar
  29. 29.
    Kowald, D., Kopeinik, S., Seitlinger, P., Ley, T., Albert, D., Trattner, C.: Refining frequency-based tag reuse predictions by means of time and semantic context. In: Atzmueller, M., Chin, A., Scholz, C., Trattner, C. (eds.) MUSE/MSM 2013, LNAI 8940. LNCS, vol. 8940, pp. 55–74. Springer, Heidelberg (2015)Google Scholar
  30. 30.
    Anderson, J.R., Schooler, L.J.: Reflections of the environment in memory. Psychol. Sci. 2(6), 396–408 (1991)CrossRefGoogle Scholar
  31. 31.
    Love, B.C., Medin, D.L., Gureckis, T.M.: Sustain: a network model of category learning. Psychol. Rev. 111(2), 309 (2004)CrossRefGoogle Scholar
  32. 32.
    Benchmark folksonomy data from bibsonomy, Knowledge and Data Engineering Group. University of Kassel, 2013/2015.
  33. 33.
    Stefaner, M., Dalla Vecchia, E., Condotta, M., Wolpers, M., Specht, M., Apelt, S., Duval, E.: MACE – enriching architectural learning objects for experience multiplication. In: Duval, E., Klamma, R., Wolpers, M. (eds.) EC-TEL 2007. LNCS, vol. 4753, pp. 322–336. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  34. 34.
    Vuorikari, R., Massart, D.: Datatel challenge: European schoolnet’s travel well dataset. In: Proceedings of RecSysTEL 2010 (2010)Google Scholar
  35. 35.
    Beham, G., Stern, H., Lindstaedt, S.: Aposdle-ds a dataset from the Aposdle work integrated learning system. In: Proceedings of RecSysTEL 2010 (2010)Google Scholar
  36. 36.
    Marinho, L.B., Hotho, A., Jäschke, R., Nanopoulos, A., Rendle, S., Schmidt-Thieme, L., Stumme, G., Symeonidis, P.: Recommender Systems for Social Tagging Systems. Springer, New York (2012)CrossRefGoogle Scholar
  37. 37.
    Sakai, T.: On the reliability of information retrieval metrics based on graded relevance. Inf. Process. Manage. 43(2), 531–548 (2007)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Simone Kopeinik
    • 1
    Email author
  • Dominik Kowald
    • 1
  • Elisabeth Lex
    • 1
  1. 1.Knowledge Technologies InstituteGraz University of TechnologyGrazAustria

Personalised recommendations