Applied Intelligence

, Volume 47, Issue 2, pp 277–290 | Cite as

A recommendation approach for programming online judges supported by data preprocessing techniques

  • Raciel Yera
  • Luis Martínez


The use of programming online judges (POJ) to support students acquiring programming skills is common nowadays because this type of software contains a large collection of programming exercises to be solved by students. A POJ not only provides exercises but also automates the code compilation and its evaluation process. A common problem that students face when using POJ is information overload, as choosing the right problem to solve can be quite frustrating due to the large number of problems offered. The integration of current POJs into e-learning systems such as Intelligent Tutoring Systems (ITSs) is hard because of the lack of necessary information in ITSs. Hence, the aim of this paper is to support students with the information overload problem by using a collaborative filtering recommendation approach that filters out programming problems suitable for students’ programming skills. It uses an enriched user-problem matrix that implies a better student role representation, facilitating the computation of closer neighborhoods and hence a more accurate recommendation. Additionally a novel data preprocessing step that manages anomalous users’ behaviors that could affect the recommendation generation is also integrated in the recommendation process. A case study is carried out on a POJ real dataset showing that the proposal outperforms other previous approaches.


Programming online judges Recommender systems Collaborative filtering Users’ inconsistencies 



This research work was partially supported by the Research Project TIN2015-66524-P.


  1. 1.
    Adomavicius G, Tuzhilin AT (2005) Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng 17(6):734–749CrossRefGoogle Scholar
  2. 2.
    Ala-Mutka KM (2005) A survey of automated assessment approaches for programming assignments. Comput Sci Educ 15(2):83–102CrossRefGoogle Scholar
  3. 3.
    Amatriain X, Pujol JM (2015) Data mining methods for recommender systems. In: Recommender Systems Handbook. Springer, pp 227–262Google Scholar
  4. 4.
    Amatriain X, Pujol JM, Oliver N (2009a) I like it... i like it not: Evaluating user ratings noise in recommender systems. In: User modeling, adaptation, and personalization. Springer, pp 247–258Google Scholar
  5. 5.
    Amatriain X, Pujol JM, Tintarev N (2009b) Rate it again: increasing recommendation accuracy by user re-rating. In: Proceedings of the third ACM conference on Recommender systems. ACM, pp 173–180Google Scholar
  6. 6.
    Arefin AS (2006) Art of Programming Contest. Gyankosh ProkashoniaGoogle Scholar
  7. 7.
    Caiza J, Del Amo J (2013) Programming assignments automatic grading: Review of tools and implementations. In: Proceedings of INTED, vol 2013, pp 5691–5700Google Scholar
  8. 8.
    Campos PG, Díez F, Cantador I (2014) Time-aware recommender systems: a comprehensive survey and analysis of existing evaluation protocols. User Model User-Adap Inter 24(1-2):67–119CrossRefGoogle Scholar
  9. 9.
    Castro J, Yera Toledo R, Martínez L (2016) An empirical study of natural noise management in group recommendation systems. Decision Support Systems. doi: 10.1016/j.dss.2016.09.020
  10. 10.
    Chen LC, Kuo PJ, Liao IE (2015) Ontology-based library recommender system using mapreduce. Clust Comput 18(1):113–121CrossRefGoogle Scholar
  11. 11.
    Christensen I, Schiaffino S (2014) Social influence in group recommender systems. Online Inf Rev 38 (4):524–542CrossRefGoogle Scholar
  12. 12.
    Dascalu MI, Bodea CN, Mihailescu MN, Tanase EA, Ordoñez de Pablos P (2016) Educational recommender systems and their application in lifelong learning. Behav Inform Technol 35(4):290–297CrossRefGoogle Scholar
  13. 13.
    De Maio C, Fenza G, Gaeta M, Loia V, Orciuoli F, Senatore S (2012) Rss-based e-learning recommendations exploiting fuzzy fca for knowledge modeling. Appl Soft Comput 12(1):113–124CrossRefGoogle Scholar
  14. 14.
    De Oliveira MG, Ciarelli PM, Oliveira E (2013) Recommendation of programming activities by multi-label classification for a formative assessment of students. Expert Syst Appl 40(16):6641–6651CrossRefGoogle Scholar
  15. 15.
    Derrac J, García S, Molina D, Herrera F (2011) A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evol Comput 1(1):3–18CrossRefGoogle Scholar
  16. 16.
    Drachsler H, Verbert K, Santos OC, Manouselis N (2015) Panorama of recommender systems to support learning. In: Recommender systems handbook. Springer, pp 421–451Google Scholar
  17. 17.
    Fenza G, Orciuoli F (2016) Building pedagogical models by formal concept analysis. In: International Conference on Intelligent Tutoring Systems. Springer, pp 144–153Google Scholar
  18. 18.
    Gaeta M, Orciuoli F, Paolozzi S, Salerno S (2011) Ontology extraction for knowledge reuse: The e-learning perspective. IEEE Trans Syst Man Cybern Part A Syst Humans 41(4):798–809CrossRefGoogle Scholar
  19. 19.
    Gruber TR (1995) Toward principles for the design of ontologies used for knowledge sharing? Int J Hum Comput Stud 43(5):907–928CrossRefGoogle Scholar
  20. 20.
    Guarino N, Giaretta P (1995) Ontologies and knowledge bases towards a terminological clarification. In: Towards very large knowledge bases: knowledge building & knowledge sharing. IOS Press, pp 25–32Google Scholar
  21. 21.
    Gunawardana A, Shani G (2009) A Survey of Accuracy Evaluation Metrics of Recommendation Tasks. J Mach Learn Res 10:2935–2962MathSciNetzbMATHGoogle Scholar
  22. 22.
    Hsiao IH, Sosnovsky S, Brusilovsky P (2010) Guiding students to the right questions: adaptive navigation support in an e-learning system for java programming. J Comput Assist Learn 26(4):270–283CrossRefGoogle Scholar
  23. 23.
    Huang Z, Zeng D, Chen H (2007) A comparison of collaborative-filtering recommendation algorithms for e-commerce. IEEE Intell Syst 5:68–78CrossRefGoogle Scholar
  24. 24.
    Klaṡnja-Milićević A, Vesin B, Ivanović M, Budimac Z (2011) E-learning personalization based on hybrid recommendation strategy and learning style identification. Comput Educ 56(3):885–899CrossRefGoogle Scholar
  25. 25.
    Klaṡnja-Milićević A, Ivanović M, Nanopoulos A (2015) Recommender systems in e-learning environments: a survey of the state-of-the-art and possible extensions. Artif Intell Rev 44(4):571–604CrossRefGoogle Scholar
  26. 26.
    Krishnan S, Patel J, Franklin MJ, Goldberg K (2014) A methodology for learning, analyzing, and mitigating social influence bias in recommender systems. In: Proceedings of the 8th ACM Conference on Recommender systems. ACM, pp 137–144Google Scholar
  27. 27.
    Kurnia A, Lim A, Cheang B (2001) Online judge. Comput Educ 36(4):299–315CrossRefGoogle Scholar
  28. 28.
    Leal JP, Silva F (2003) Mooshak: a web-based multi-site programming contest system. Software: Practice and Experience 33(6):567–581Google Scholar
  29. 29.
    Li B, Chen L, Zhu X, Zhang C (2013) Noisy but non-Malicious user detection in social recommender systems. World Wide Web 16(5-6):677–699CrossRefGoogle Scholar
  30. 30.
    Llana L, Martin-Martin E, Pareja-Flores C, Velázquez-Iturbide JA ́ (2014) Flop: a user-friendly system for automated program assessment. Journal of Universal Computer Science 20(9):1304–1326Google Scholar
  31. 31.
    Lops P, Gemmis M, Semeraro G (2011) Content-based recommender systems: State of the art and trends. In: Recommender systems handbook springer US, chap, vol 3, pp 73–105Google Scholar
  32. 32.
    Lu J, Wu D, Mao M, Wang W, Zhang G (2015) Recommender system application developments: a survey. Decis Support Syst 74:12–32CrossRefGoogle Scholar
  33. 33.
    Martínez L, Pérez L G, Barranco M (2007) A multigranular linguistic content-based recommendation model. Int J Intell Syst 22(5):419–434CrossRefGoogle Scholar
  34. 34.
    Martínez L, Barranco MJ, Pérez LG, Espinilla M (2008) A knowledge based recommender system with multigranular linguistic information. International Journal of Computational Intelligence Systems 1(3):225–236CrossRefGoogle Scholar
  35. 35.
    Miranda S, Orciuoli F, Sampson DG (2016) A skos-based framework for subject ontologies to improve learning experiences. Comput Hum Behav 61:609–621CrossRefGoogle Scholar
  36. 36.
    Murray T (1999) Authoring intelligent tutoring systems: an analysis of the state of the art. Int J Artif Intell Educ 10:98–129Google Scholar
  37. 37.
    Nadolski RJ, Van den Berg B, Berlanga AJ, Drachsler H, Hummel HG, Koper R, Sloep PB (2009) Simulating light-weight personalised recommender systems in learning networks: a case for pedagogy-oriented and rating-based hybrid recommendation strategies. Journal of Artificial Societies and Social Simulation 12(1):4Google Scholar
  38. 38.
    Ning X, Desrosiers C, Karypis G (2015) A comprehensive survey of neighborhood-based recommendation methods. In: Recommender Systems Handbook. Springer, pp 37–76Google Scholar
  39. 39.
    Noguera J, Barranco M, Segura R, Martínez L (2012) A mobile 3d-gis hybrid recommender system for tourism. Inf Sci 215:37– 52CrossRefGoogle Scholar
  40. 40.
    O’Mahony MP, Hurley NJ, Silvestre G (2006) Detecting noise in recommender system databases. In: Proceedings of the 11th international conference on Intelligent user interfaces. ACM, pp 109–115Google Scholar
  41. 41.
    Petit J, Giménez O, Roura S (2012) Jutge. org: an educational programming judge. In: Proceedings of the 43rd ACM technical symposium on Computer Science Education. ACM, pp 445– 450Google Scholar
  42. 42.
    Pham HX, Jung JJ (2013) Preference-based user rating correction process for interactive recommendation systems. Multimedia tools and applications 65(1):119–132CrossRefGoogle Scholar
  43. 43.
    Piramuthu S, Kapoor G, Zhou W, Mauw S (2012) Input online review data and related bias in recommender systems. Decis Support Syst 53(3):418–424CrossRefGoogle Scholar
  44. 44.
    Polson MC, Richardson JJ (2013) Foundations of intelligent tutoring systems. Psychology PressGoogle Scholar
  45. 45.
    Regueras LM, Verdú E, Muňoz MF, Pérez MA, De Castro JP, Verdú MJ (2009) Effects of competitive e-learning tools on higher education students: a case study. IEEE Trans Educ 52(2):279–285CrossRefGoogle Scholar
  46. 46.
    Ricci F (2015) Recommender systems handbook. Springer, Shapira BCrossRefzbMATHGoogle Scholar
  47. 47.
    Ruiz-Iniesta A, Jimenez-Diaz G, Gomez-Albarran M (2014) A semantically enriched context-aware oer recommendation strategy and its application to a computer science oer repository. IEEE Trans Educ 57(4):255–260CrossRefGoogle Scholar
  48. 48.
    Said A, Jain BJ, Narr S, Plumbaum T (2012) Users and noise: The magic barrier of recommender systems. In: User modeling, Adaptation, and Personalization. Springer, pp 237–248Google Scholar
  49. 49.
    Skiena SS (2006) Revilla MA. The programming contest training manual. Springer Science & Business Media, Programming challengesGoogle Scholar
  50. 50.
    Sleeman D, Brown JS (1982) Intelligent tutoring systems. Academic Press, LondonGoogle Scholar
  51. 51.
    Thai-Nghe N, Drumond L, Horváth T, Nanopoulos A, Schmidt-Thieme L (2011) Matrix and tensor factorization for predicting student performance. In: Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU), pp 69–78Google Scholar
  52. 52.
    Verdú E, Regueras LM, Verdú MJ, Leal JP, de Castro JP, Queirós R (2012) A distributed system for learning programming on-line. Comput Educ 58(1):1–10Google Scholar
  53. 53.
    Vesin B, Klaṡnja-Milićević A, Ivanović M, Budimac Z (2013) Applying recommender systems and adaptive hypermedia for e-learning personalization. Computing and Informatics 32(3):629–659Google Scholar
  54. 54.
    Wang GP, Chen SY, Yang X, Feng R (2016) Ojpot: online judge & practice oriented teaching idea in programming courses. Eur J Eng Educ 41(3):304–319CrossRefGoogle Scholar
  55. 55.
    Wang PY, Yang HC (2012) Using collaborative filtering to support college students’ use of online forum for english learning. Comput Educ 59(2):628–637CrossRefGoogle Scholar
  56. 56.
    Winoto P, Tang TY, McCalla GI (2012) Contexts in a paper recommendation system with collaborative filtering. The International Review of Research in Open and Distributed Learning 13(5):56–75CrossRefGoogle Scholar
  57. 57.
    Yera Toledo R, Caballero Mota Y (2014) An e-learning collaborative filtering approach to suggest problems to solve in programming online judges. International Journal of Distance Education Technologies 12(2):51–65CrossRefGoogle Scholar
  58. 58.
    Yera Toledo R, Caballero Mota Y, Garcia Borroto M (2013) A regularity-based preprocessing method for collaborative recommender systems. J Inf Process Syst 9(3):435– 460CrossRefGoogle Scholar
  59. 59.
    Yera Toledo R, Caballero Mota Y, Martínez L (2015) Correcting noisy ratings in collaborative recommender systems. Knowl-Based Syst 76:96–108CrossRefGoogle Scholar
  60. 60.
    Yera Toledo R, Castro J, Martínez L (2016) A fuzzy model for managing natural noise in recommender systems. Appl Soft Comput 40:187–198CrossRefGoogle Scholar
  61. 61.
    Yu R, Cai Z, Du X, He M, Wang Z, Yang B, Chang P (2015) The research of the recommendation algorithm in online learning. International Journal of Multimedia and Ubiquitous Engineering 10(4):71–80CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.University of Ciego de ÁvilaCiego de ÁvilaCuba
  2. 2.Computer Science DepartmentUniversity of JaénJaénSpain

Personalised recommendations