Comparison of Feedback Strategies for Supporting Programming Learning in Integrated Development Environments (IDEs)

  • Jarno CoenenEmail author
  • Sebastian Gross
  • Niels Pinkwart
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 629)


In this paper we investigate whether providing feedback to learners within an Integrated Development Environment (IDE) helps them write correct programs. Here, we use two approaches: feedback based on stack trace analysis, and feedback based on structural comparisons of a learner program and appropriate sample programs. In order to investigate both approaches, we developed two prototypical extensions for the Eclipse IDE. In a laboratory study, we empirically evaluated the impact of the extensions on learners’ performance while they solved programming tasks. The statistical analyses did not reveal any statistically significant effects of the prototype extensions on the performance of the learners, however, the results of a qualitative analysis imply that the provided feedback had at least a marginal impact on the performance of some learners. Also, feedback from the participants confirmed the benefit of providing feedback directly within IDEs.


Adaptive feedback Integrated Development Environment Java programming 



This work was supported by the Deutsche Forschungsgemeinschaft (DFG) under the grant “DynaFIT – Learning Dynamic Feedback in Intelligent Tutoring Systems” (PI 764/6-2).


  1. 1.
    European Commission: Coding - the 21st century skill 6 July 2016. Accessed 4 Nov 2016
  2. 2.
    Balanskat, A., Engelhardt, K.: Computing Our Future - Computer Programming and Coding - Priorities, School Curricula and Initiatives Across Europe. European Schoolnet, Brussels (2015)Google Scholar
  3. 3.
    The White House - Office of the Press Secretary: Remarks of President Barack Obama – State of the Union Address as Delivered, 13 January 2016. Accessed 04 Nov 2016
  4. 4.
    Zweben, S., Bizot, B.: 2015 taulbee survey: continued booming undergraduate CS enrollment; doctoral degree production dips slightly. Comput. Res. News 28(5), 2–60 (2016)Google Scholar
  5. 5.
    Self, J.: The defining characteristics of intelligent tutoring systems research: ITS care, precisely. Int. J. Artif. Intell. Educ. (IJAIED) 10, 350–364 (1998)Google Scholar
  6. 6.
    Nienaltowski, M., Pedroni, M., Meyer, B.: Compiler error messages: what can help novices? In: Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education, pp. 168–172 (2008)Google Scholar
  7. 7.
    Marceau, G., Fisler, K., Krishnamurthi, S.: Mind your language: on novices’ interactions with error messages. In: Proceedings of the 10th SIGPLAN Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward! 2011), pp. 3–18 (2011)Google Scholar
  8. 8.
    Hristova, M., Misra, A., Rutter, M., Mercuri, R.: Identifying and correcting Java programming errors for introductory computer science students. In: Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education (SIGCSE 2003), pp. 153–156 (2003)Google Scholar
  9. 9.
    Murphy, C., Kim, E., Kaiser, G., Cannon, A.: Backstop: a tool for debugging runtime errors. In: Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education (SIGCSE 2008), pp. 173–177 (2008)Google Scholar
  10. 10.
    Hartmann, B., MacDougall, D., Brandt, J., Klemmer, S.: What would other programmers do: suggesting solutions to error messages. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2010), pp. 1019–1028 (2010)Google Scholar
  11. 11.
    Flowers, T., Carver, C.A., Jackson, J.: Empowering students and building confidence in novice programmers through Gauntlet. In: 34th Annual Frontiers in Education, FIE 2004, pp. T3H10–T3H13 (2004)Google Scholar
  12. 12.
    Holland, J., Mitrovic, A., Martin, B.: J-LATTE: a constraint-based tutor for Java. In: Proceedings of the 17th International Conference on Computers in Education, pp. 142–146 (2009)Google Scholar
  13. 13.
    Sykes, E.: Design, development and evaluation of the Java intelligent tutoring system. Tech. Inst. Cogn. Learn. 8, 25–65 (2010)Google Scholar
  14. 14.
    Abu-Naser, S., Ahmed, A., Al-Masri, N., Deeb, A., Moshtaha, E., Abu-Lamdy, M.: An intelligent tutoring system for learning Java objects. Int. J. Artif. Intell. Appl. (IJAIA) 2(2), 68–77 (2011)Google Scholar
  15. 15.
    Codeanywhere Inc.: Most Popular Desktop IDEs & Code Editors in 2014, 13 January 2015. Accessed 04 Nov 2016
  16. 16.
    Maple, S.: Java Tools and Technologies Landscape Report 2016, 14 July 2016. Accessed 4 Nov 2016
  17. 17.
    Biradar, M.: Popularity of Programming Languages, 28 July 2015. Accessed 4 Nov 2016
  18. 18.
    Stack Exchange Inc.: Developer Survey Results 2016, March 2016. Accessed 5 Nov 2016
  19. 19.
    Jackson, J., Cobb, M., Carver, C.: Identifying top Java errors for novice programmers. In: Proceedings - Frontiers in Education Conference, p. T4C (2005)Google Scholar
  20. 20.
    Jadud, J.: A first look at novice compilation behaviour using BlueJ. Comput. Sci. Educ. 15, 25–40 (2005)CrossRefGoogle Scholar
  21. 21.
    Altadmri, A., Brown, N.: 37 million compilations: investigating novice programming mistakes in large-scale student data. In: Proceedings of the 46th ACM Technical Symposium on Computer Science Education, pp. 522–527 (2015)Google Scholar
  22. 22.
    Tabanao, E., Rodrigo, M., Jadud, M.: Predicting at-risk novice Java programmers through the analysis of online protocols. In: Proceedings of the Seventh International Workshop on Computing Education Research (ICER 2011), pp. 85–92 (2011)Google Scholar
  23. 23.
    McCall, D., Kölling, M.: Meaningful categorisation of novice programmer errors. In: Frontiers in Education Conference, pp. 2589–2596 (2014)Google Scholar
  24. 24.
    Becker, B.: An effective approach to enhancing compiler error messages. In: Proceedings of the 47th ACM Technical Symposium on Computing Science Education (SIGCSE 2016), pp. 126–131 (2016)Google Scholar
  25. 25.
    Denny, P., Luxton-Reilly, A., Tempero, E.: All syntax errors are not equal. In: Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education (ITiCSE 2012), pp. 75–80 (2012)Google Scholar
  26. 26.
    Gross, S., Mokbel, B., Hammer, B., Pinkwart, N.: Learning feedback in intelligent tutoring systems. KI - Künstliche Intell. 29(4), 413–418 (2015)CrossRefGoogle Scholar
  27. 27.
    Gross, S., Pinkwart, N.: How do learners behave in help-seeking when given a choice? Int. J. Artif. Intell. Educ. 9112, 600–603 (2015)CrossRefGoogle Scholar
  28. 28.
    Mokbel, B., Gross, S., Paassen, B., Pinkwart, N., Hammer, B.: Domain-independent proximity measures in intelligent tutoring systems. In: Proceedings of the 6th International Conference on Educational Data Mining (EDM), pp. 334–335 (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Humboldt-Universität zu BerlinBerlinGermany

Personalised recommendations