Ask-Elle: an Adaptable Programming Tutor for Haskell Giving Automated Feedback

  • Alex Gerdes
  • Bastiaan Heeren
  • Johan Jeuring
  • L. Thomas van Binsbergen


Ask-Elle is a tutor for learning the higher-order, strongly-typed functional programming language Haskell. It supports the stepwise development of Haskell programs by verifying the correctness of incomplete programs, and by providing hints. Programming exercises are added to Ask-Elle by providing a task description for the exercise, one or more model solutions, and properties that a solution should satisfy. The properties and model solutions can be annotated with feedback messages, and the amount of flexibility that is allowed in student solutions can be adjusted. The main contribution of our work is the design of a tutor that combines (1) the incremental development of different solutions in various forms to a programming exercise with (2) automated feedback and (3) teacher-specified programming exercises, solutions, and properties. The main functionality is obtained by means of strategy-based model tracing and property-based testing. We have tested the feasibility of our approach in several experiments, in which we analyse both intermediate and final student solutions to programming exercises, amongst others.


Functional programming Haskell Tutoring Model tracing Automated feedback Adaptability 



The authors would like to thank the anonymous reviewers for their helpful suggestions. We also thank Andres Löh, Doaitse Swierstra, and Jurriaan Hage for allowing us to perform experiments with our tutor in their classes. For the assessment experiment, Stefan Holdermans provided a set of suitable programming exercises, accompanied with a large amount of corrected student solutions. Bram Vaessen analysed the scores of our questionnaire. Peter van de Werken, Bram Schuur, Tom Tervoort and Gabe Dijkstra contributed to the source code of Ask-Elle. Hieke Keuning found some of the related work, and is working on a systematic literature review on automated feedback for programming exercises.


  1. Ala-Mutka, K. M. (2005). A survey of automated assessment approaches for programming assignments. Computer Science Education, 15(2), 83–102.CrossRefGoogle Scholar
  2. Anderson, J. R., Conrad, F. G., & Corbett, A. T. (1986). Skill acquisition and the LISP tutor. Cognitive Science, 13, 467–505.CrossRefGoogle Scholar
  3. Anderson, J. R., Corbett, A. T., Koedinger, K. R., & Pelletier, R. (1995). Cognitive tutors: lessons learned. The Journal of the Learning Sciences, 4(2), 167–207.CrossRefGoogle Scholar
  4. Barendregt, H. P. (1984). The Lambda calculus: Its syntax and semantics, revised edition, volume 103 of Studies in Logic and the Foundations of Mathematics. North-Holland.Google Scholar
  5. Bird, R. S. (1998). Introduction to functional programming using Haskell. Prentice-Hall.Google Scholar
  6. Bokhove, C., & Drijvers, P. (2010). Digital tools for algebra education: criteria and evaluation. International Journal of Computers for Mathematical Learning, 15(1), 45–62.CrossRefGoogle Scholar
  7. Brusilovsky, P., Schwarz, E., and Weber, G. (1996). Elm-art: An intelligent tutoring system on world wide web. In Frasson, C., Gauthier, G., and Lesgold, A., (eds.), Intelligent Tutoring Systems, volume 1086 of LNCS, pages 261–269. Springer-Verlag.Google Scholar
  8. Claessen, K. and Hughes, J. (2000). QuickCheck: A lightweight tool for random testing of Haskell programs. In Proceedings of ICFP 2000: the 5th ACM SIGPLAN International Conference on Functional Programming, pages 268–279. ACM.Google Scholar
  9. Corbett, A. T. and Anderson, J. R. (2001). Locus of feedback control in computer-based tutoring: Impact on learning rate, achievement and attitudes. In Proceedings of CHI’01: the SIGCHI Conference on Human Factors in Computing Systems, pages 245–252. ACM.Google Scholar
  10. Corbett, A., Anderson, J., and Patterson, E. (1988). Problem compilation and tutoring flexibility in the LISP tutor. In Intelligent Tutoring Systems, pages 423–429.Google Scholar
  11. Douce, C., Livingstone, D., and Orwell, J. (2005). Automatic test-based assessment of programming: A review. Journal on Educational Resources in Computing (JERIC), 5(3)Google Scholar
  12. Gerdes, A., Jeuring, J., and Heeren, B. (2010). Using strategies for assessment of programming exercises. In Lewandowski, G., Wolfman, S. A., Cortina, T. J., and Walker, E. L., (eds.), Proceedings of SIGCSE 2010: the 41st ACM technical symposium on Computer science education, pages 441–445. ACM.Google Scholar
  13. Gerdes, A., Heeren, B., and Jeuring, J. (2012). Teachers and students in charge — Using annotated model solutions in a functional programming tutor. In Proceedings EC-TEL 2012: the 7th European Conference of Technology Enhanced Learning, volume 7563 of LNCS, pages 383–388. Springer-Verlag.Google Scholar
  14. Gerdes, A., Jeuring, J., and Heeren, B. (2012). An interactive functional programming tutor. In T. Lapidot, J. Gal-Ezer, M. Caspersen, and O. Hazzan (eds.), Proceedings of ITICSE 2012: the 17th Annual Conference on Innovation and Technology in Computer Science Education, pages 250–255. ACM.Google Scholar
  15. Heeren, B., & Jeuring, J. (2014). Feedback services for stepwise exercises. Science of Computer Programming, Special Issue on Software Development Concerns in the e-Learning Domain, 88, 110–129.Google Scholar
  16. Heeren, B., Leijen, D., and IJzendoorn, A. V. (2003). Helium, for learning Haskell. In Proceedings of Haskell 2003: The ACM SIGPLAN Workshop on Haskell, pages 62–71. ACM.Google Scholar
  17. Heeren, B., Jeuring, J., & Gerdes, A. (2010). Specifying rewrite strategies for interactive exercises. Mathematics in Computer Science, 3(3), 349–370.zbMATHCrossRefGoogle Scholar
  18. Holland, J., Mitrovic, A., and Martin, B. (2009). J-Latte: a constraint-based tutor for Java. In Proceedings of ICCE 2009: The 17th International Conference on Computers in Education, pages 142–146.Google Scholar
  19. Hong, J. (2004). Guided programming and automated error analysis in an in telligent Prolog tutor. International Journal on Human-Computer Studies, 61(4), 505–534.CrossRefGoogle Scholar
  20. Hudak, P. (2000). The Haskell School of expression: Learning functional programming through multimedia. Cambridge University Press.Google Scholar
  21. Hutton, G. (2007). Programming in Haskell. Cambridge University Press.Google Scholar
  22. Jeuring, J., Gerdes, A., and Heeren, B. (2012). Ask-elle: A Haskell tutor — Demonstration —. In Proceedings EC-TEL 2012: the 7th European Conference of Technology Enhanced Learning, volume 7563 of LNCS, pages 453–458. Springer-Verlag.Google Scholar
  23. Jeuring, J., Binsbergen, L. T. V., Gerdes, A., and Heeren, B. (2014). Model solutions and properties for diagnosing student programs in Ask-Elle. In Barendsen, E. and Dagiené, V., editors, Proceedings of CSERC 2014: Computer Science Education Research Conference, pages 31–40. ACM.Google Scholar
  24. Jin, W., Barnes, T., Stamper, J. C., Eagle, M. J., Johnson, M., and Lehmann, L. (2012). Program representation for automatic hint generation for a data-driven novice programming tutor. In S. A. Cerri, W. J. Clancey, G. Papadourakis, and K. Panourgia (eds.), Proceedings of ITS 2012: the 11th International Conference on Intelligent Tutoring Systems, volume 7315 of LNCS, pages 304–309. Springer-Verlag.Google Scholar
  25. Johnson, W. L. (1990). Understanding and debugging novice programs. Artificial Intelligence, 42(1), 51–97.CrossRefGoogle Scholar
  26. Johnson, W. L., & Soloway, E. (1985). Proust: knowledge-based program understanding. IEEE Transactions on Software Engineering, 11(3), 267–275.CrossRefGoogle Scholar
  27. Joint Task Force on Computing Curricula Association for Computing Machinery (ACM) and IEEE Computer Society (2013). Computer Science Curricula 2013: Curriculum Guidelines for Undergraduate Degree Programs in Computer Sci ence. ACM.Google Scholar
  28. Keuning, H., Heeren, B., and Jeuring, J. (2014). Strategy-based feedback in a programming tutor. In E. Barendsen, and V. Dagiené, (eds.), Proceedings of CSERC 2014: Computer Science Education Research Conference, pages 43–54. ACM.Google Scholar
  29. Kölling, M., Quig, B., Patterson, A., & Rosenberg, J. (2003). The BlueJ system and its pedagogy. Journal of Computer Science Education, Special issue on Learning and Teaching Object Technology, 13(4), 249–268.Google Scholar
  30. Kumar, A. N. (2008). The effect of using problem-solving software tutors on the self-confidence of female students. In Proceedings of SIGCSE 2008: the 39th SIGCSE technical symposium on Computer science education, pages 523–527. ACM.Google Scholar
  31. Le, N.-T., & Menzel, W. (2009). Using weighted constraints to diagnose errors in logic programming - the case of an ill-defined domain. International Journal of Artificial Intelligence in Education, 19, 381–400.Google Scholar
  32. Le, N.-T. and Pinkwart, N. (2014). Towards a classification for programming exercises. In Proceedings of the 2nd Workshop on AI-supported Education for Computer Science.Google Scholar
  33. Le, N.-T., Menzel, W., and Pinkwart, N. (2009). Evaluation of a constraint-based homework assistance system for logic programming. In Proceedings of ICCE 2009: the 17th International Conference on Computers in Education, pages 51–58.Google Scholar
  34. Le, N.-T., Strickroth, S., Gross, S., and Pinkwart, N. (2013). A review of AI-supported tutoring approaches for learning programming. In Advanced Computational Methods for Knowledge Engineering, volume 479 of Studies in Computational Intelligence, pages 267–279. Springer-Verlag.Google Scholar
  35. Looi, C.-K. (1991). Automatic debugging of Prolog programs in a Prolog intelligent tutoring system. Instructional Science, 20, 215–263.CrossRefGoogle Scholar
  36. Meyer, B. (1992). Applying “design by contract”. Computer, 25(10), 40–51.CrossRefGoogle Scholar
  37. Mol, M. D., Eekelen, M. V., and Plasmeijer, R. (2002). Theorem proving for functional programmers - Sparkle: a functional theorem prover. In Proceedings of IFL 2001, 13th International Workshop onImplementation of Functional Languages, volume 2312 of LNCS, pages 55–72. Springer-Verlag.Google Scholar
  38. Mory, E. (2003). Feedback research revisited. In D. Jonassen, (ed.), Handbook of research for educational communications and technology, pages 745–783.Google Scholar
  39. Nwana, H. S. (1990). Intelligent tutoring systems: an overview. Artificial Intelligence Review, 4(4), 251–277.CrossRefGoogle Scholar
  40. Odekirk-Hash, E. and Zachary, J. L. (2001). Automated feedback on programs means students need less help from teachers. In Proceedings of SIGCSE 2001: the 32nd SIGCSE technical symposium on Computer Science Education, pages 55–59. ACM.Google Scholar
  41. Peyton Jones, S. (2003). Haskell 98, Language and Libraries. The Revised Report. Cambridge University Press.Google Scholar
  42. Price, T. W. and Barnes, T. (2015). Creating data-driven feedback for novices in goal-driven programming projects. In C. Conati, N. Heffernan, A. Mitrovic, and M. F. Verdejo (eds.), Artificial Intelligence in Education, volume 9112 of LNCS, pages 856–859. Springer-Verlag.Google Scholar
  43. Rivers, K. and Koedinger, K. R. (2013). Automatic generation of programming feedback: A data-driven approach. In Proceedings of AIEDCS 2013: the First Workshop on AI-supported Education for Computer Science.Google Scholar
  44. Rivers, K. and Koedinger, K. R. (2014). Automating hint generation with solution space path construction. In S. Trausan-Matu, K. E. Boyer, M. Crosby, and K. Panourgia, (eds.), Proceedings of ITS 2014: the 12th International Conference on Intelligent Tutoring Systems, volume 8474 of LNCS, pages 329–339. Springer-Verlag.Google Scholar
  45. Sison, R. C., Numao, M., & Shimura, M. (2000). Multistrategy discovery and detection of novice programmer errors. Machine Learning, 38, 157–180.CrossRefGoogle Scholar
  46. Soloway, E. (1985). From problems to programs via plans: the content and structure of knowledge for introductory LISP programming. Journal of Educational Computing Research, 1(2), 157–172.CrossRefGoogle Scholar
  47. Striewe, M. and Goedicke, M. (2014). A review of static analysis approaches for programming exercises. In Computer Assisted Assessment. Research into E-Assessment, volume 439 of Communications in Computer and Information Science, pages 100–113. Springer-Verlag.Google Scholar
  48. Sweller, J., Ayres, P., and Kalyuga, S. (2011). Cognitive load theory. Springer-Verlag.Google Scholar
  49. Swierstra, S. D. and Alcocer, P. R. A. (1999). Fast, error correcting parser combinators: A short tutorial. In SOFSEM99: Theory and Practice of Informatics, pages 112–131. Springer-Verlag.Google Scholar
  50. Sykes, E. and Franek, F. (2004). A prototype for an intelligent tutoring system for students learning to program in Java. Advanced Technology for Learning, 1(1).Google Scholar
  51. Thompson, S. (2011). Haskell: The craft of functional programming. Addison-Wesley.Google Scholar
  52. Thorburn, G., & Rowe, G. (1997). Pass: an automated system for program assessment. Computers & Education, 29(4), 195–206.CrossRefGoogle Scholar
  53. Truong, N., Roe, P., and Bancroft, P. (2004). Static analysis of students’ Java rograms. In Proceedings of ACE’04: the sixth conference on Australasian computing education, pages 317–325. Australian Computer Society, Inc.Google Scholar
  54. Ulloa, M. (1980). Teaching and learning computer programming: a survey of student problems, teaching methods, and automated instructional tools. ACM SIGCSE Bulletin, 12(2), 48–64.CrossRefGoogle Scholar
  55. VanLehn, K. (2006). The behavior of tutoring systems. International Journal on Artificial Intelligence in Education, 16(3), 227–265.Google Scholar
  56. VanLehn, K. (2011). The relative effectiveness of human tutoring, intelligent tutoring systems, and other tutoring systems. Educational Psychologist, 46(4), 197–221.CrossRefGoogle Scholar
  57. Vanneste, P., Bertels, K., & Decker, B. d. (1996). The use of reverse engineering to analyse student computer programs. Instructional Science, 24, 197–221.CrossRefGoogle Scholar
  58. Voeten, J. (2001). On the fundamental limitations of transformational design. ACM Transactions on Design Automation of Electronic Systems, 6(4), 533–552.CrossRefGoogle Scholar
  59. Wang, T., Su, X., Wang, Y., & Ma, P. (2007). Semantic similarity-based grading of student programs. Information and Software Technology, 49(2), 99–107.CrossRefGoogle Scholar
  60. Wang, T., Su, X., Ma, P., Wang, Y., & Wang, K. (2011). Ability-training-oriented automated assessment in introductory programming course. Computers & Education, 56(1), 220–226.CrossRefGoogle Scholar
  61. Xu, S., & Chee, Y. S. (2003). Transformation-based diagnosis of student programs for programming tutoring systems. IEEE Transactions on Software Engineering, 29(4), 360–384.CrossRefGoogle Scholar

Copyright information

© International Artificial Intelligence in Education Society 2016

Authors and Affiliations

  • Alex Gerdes
    • 1
  • Bastiaan Heeren
    • 2
  • Johan Jeuring
    • 3
  • L. Thomas van Binsbergen
    • 4
  1. 1.Chalmers University of TechnologyGothenburgSweden
  2. 2.Open Universiteit NederlandHeerlenNetherlands
  3. 3.Utrecht University and Open Universiteit NederlandUtrechtNetherlands
  4. 4.Royal Holloway, University of LondonEghamUK

Personalised recommendations