A Recommender System for Supporting Students in Programming Online Judges

  • Raciel Yera
  • Rosa M. RodríguezEmail author
  • Jorge Castro
  • Luis Martínez
Conference paper
Part of the Smart Innovation, Systems and Technologies book series (SIST, volume 75)


Programming Online Judges (POJs) are tools that contain a large collection of programming problems to be solved by students as a component of their training and programming practices. This contribution presents a recommendation approach to suggest to students the more suitable problems to solve for increasing their performance and motivation in POJs. Some key features of the approach are the use of an enriched user-problem matrix that incorporates specific information related to the user performance in the POJ, and the development of a strategy for natural noise management in such a matrix. The experimental evaluation shows the improvements of the proposal as compared to previous works.


Programming online judges Problems recommendation Natural noise management 



This research work was partially supported by the Spanish National research project TIN2015-66524-P, the Spanish Ministry of Economy and Finance Postdoctoral Fellow (IJCI-2015-23715), the Spanish FPU fellowship (FPU13/01151) and ERDF.


  1. 1.
    Adomavicius, G., Tuzhilin, A.: Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans. Knowl. Data Eng. 17(6), 734–749 (2005)CrossRefGoogle Scholar
  2. 2.
    Amatriain, X., Pujol, J.M.: Data mining methods for recommender systems. In: Recommender Systems Handbook, pp. 227–262. Springer (2015)Google Scholar
  3. 3.
    Caiza, J., Del Amo, J.: Programming assignments automatic grading: review of tools and implementations. In: Proceedings of INTED 2013, pp. 5691–5700 (2013)Google Scholar
  4. 4.
    Castro, J., Toledo, R.Y., Martínez, L.: An empirical study of natural noise management in group recommendation systems. Decis. Support Syst. 94, 1–11 (2016)CrossRefGoogle Scholar
  5. 5.
    Gunawardana, A., Shani, G.: A survey of accuracy evaluation metrics of recommendation tasks. J. Mach. Learn. Res. 10, 2935–2962 (2009)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Hsiao, I.-H., Sosnovsky, S., Brusilovsky, P.: Guiding students to the right questions: adaptive navigation support in an e-learning system for java programming. J. Comput. Assist. Learn. 26(4), 270–283 (2010)CrossRefGoogle Scholar
  7. 7.
    Klašnja-Milićević, A., Ivanović, M., Nanopoulos, A.: Recommender systems in e-learning environments: a survey of the state-of-the-art and possible extensions. Artif. Intell. Rev. 44(4), 571–604 (2015)CrossRefGoogle Scholar
  8. 8.
    Kurnia, A., Lim, A., Cheang, B.: Online judge. Comput. Educ. 36(4), 299–315 (2001)CrossRefGoogle Scholar
  9. 9.
    Leal, J.P., Silva, F.: Mooshak: a web-based multi-site programming contest system. Softw. Pract. Experience 33(6), 567–581 (2003)CrossRefGoogle Scholar
  10. 10.
    De Oliveira, M.G., Ciarelli, M.P., Oliveira, E.: Recommendation of programming activities by multi-label classification for a formative assessment of students. Expert Syst. Appl. 40(16), 6641–6651 (2013)CrossRefGoogle Scholar
  11. 11.
    Ruiz-Iniesta, A., Jimenez-Diaz, G., Gomez-Albarran, M.: A semantically enriched context-aware OER recommendation strategy and its application to a computer science oer repository. IEEE Trans. Educ. 57(4), 255–260 (2014)CrossRefGoogle Scholar
  12. 12.
    Toledo, R.Y., Castro, J., Martínez, L.: A fuzzy model for managing natural noise in recommender systems. Appl. Soft Comput. 40, 187–198 (2016)CrossRefGoogle Scholar
  13. 13.
    Toledo, R.Y., Mota, Y.C.: An e-learning collaborative filtering approach to suggest problems to solve in programming online judges. Int. J. Distance Educ. Technol. 12(2), 51–65 (2014)CrossRefGoogle Scholar
  14. 14.
    Toledo, R.Y., Mota, Y.C., Martínez, L.: Correcting noisy ratings in collaborative recommender systems. Knowl. Based Syst. 76, 96–108 (2015)CrossRefGoogle Scholar
  15. 15.
    Verdú, E., Regueras, L.M., Verdú, M.J., Leal, J.P., de Castro, J.P., Queirós, R.: A distributed system for learning programming on-line. Comput. Educ. 58(1), 1–10 (2012)CrossRefGoogle Scholar
  16. 16.
    Vesin, B., Klašnja-Milićević, A., Ivanović, M., Budimac, Z.: Applying recommender systems and adaptive hypermedia for e-learning personalization. Comput. Inform. 32(3), 629–659 (2013)Google Scholar
  17. 17.
    Wang, G.P., Chen, S.Y., Yang, X., Feng, R.: OJPOT: online judge & practice oriented teaching idea in programming courses. Eur. J. Eng. Educ. 41(3), 304–319 (2016)CrossRefGoogle Scholar
  18. 18.
    Yu, R., Cai, Z., Du, X., He, M., Wang, Z., Yang, B., Chang, P.: The research of the recommendation algorithm in online learning. Int. J. Multimedia Ubiquit. Eng. 10(4), 71–80 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  • Raciel Yera
    • 1
  • Rosa M. Rodríguez
    • 2
    Email author
  • Jorge Castro
    • 2
  • Luis Martínez
    • 3
  1. 1.University of Ciego de ÁvilaCiego de ÁvilaCuba
  2. 2.University of GranadaGranadaSpain
  3. 3.University of JaénJaénSpain

Personalised recommendations