Improving Engagement in Program Construction Examples for Learning Python Programming


This research is focused on how to support students’ acquisition of program construction skills through worked examples. Although examples have been consistently proven to be valuable for student’s learning, the learning technology for computer science education lacks program construction examples with interactive elements that could engage students. The goal of this work is to investigate the value of the “engaging” features in programming examples. We introduce PCEX, an online tool developed to present program construction examples in an engaging fashion. We also present the results of a controlled study with a between-subject design that was conducted in a large introductory Python programming class to compare PCEX with non-interactive worked examples focused on program construction. The results of our study show the positive impact of interactive program construction examples on student’s engagement, problem-solving performance, and learning.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8


  1. 1.

    We chose start date over due date as the time reference because some students submitted assignments after the due date was passed. Therefore, using the difference between the due date and the submission date would have made the interpretation of results more difficult, as some differences would be negative.

  2. 2.

    Using all students’ data showed the same pattern of results for this analysis.


  1. Aiken, L.S, West, S.G, & Reno, R.R. (1991). Multiple regression: Testing and interpreting interactions. Sage.

  2. Atkinson, R.K, & Renkl, A. (2007). Interactive example-based learning environments: Using interactive elements to encourage effective processing of worked examples. Educational Psychology Review, 19(3), 375–386.

    Google Scholar 

  3. Atkinson, R.K, Derry, S.J, Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of educational research, 70(2), 181– 214.

    Google Scholar 

  4. Auvinen, T., Hakulinen, L., & Malmi, L. (2018). Increasing students’ awareness of their behavior in online learning environments with visualizations and achievement badges. IEEE Transactions on Learning Technologies, 8(3), 261–273.

    Google Scholar 

  5. Bathgate, M., & Schunn, C. (2017). The psychological characteristics of experiences that influence science motivation and content knowledge. International Journal of Science Education, 39(17), 2402– 2432.

    Google Scholar 

  6. Bloom, S.B. (1956). Taxonomy of educational objectives, handbook I: The cognitive domain. New York: David McKay Co Inc.

    Google Scholar 

  7. Brna, P. (1998). Searching for examples with a programming techniques editor. Journal of Computing and Information Technology, 6(1), 13–26.

    Google Scholar 

  8. Brusilovsky, P. (1994). Explanatory visualization in an educational programming environment: connecting examples with general knowledge. In: 4th international conference on human-computer interaction, EWHCI’94, vol. 876 of Lecture Notes in Computer Science, pp. 202–212. Springer-Verlag.

  9. Brusilovsky, P, & Yudelson, M.V. (2008). From webex to navex: Interactive access to annotated program examples. Proceedings of the IEEE, 96(6), 990–999.

    Google Scholar 

  10. Brusilovsky, P., Yudelson, M., & Hsiao, I-H. (2009). Problem solving examples as first class objects in educational digital libraries: Three obstacles to overcome. Journal of Educational Multimedia and Hypermedia, 18(3), 267–288.

    Google Scholar 

  11. Byrne, M. D, Catrambone, R., & Stasko, J.T. (1999). Evaluating animations as student aids in learning computer algorithms. Computers & education, 33(4), 253–278.

    Google Scholar 

  12. Catrambone, R. (1998). The subgoal learning model: Creating better examples so that students can solve novel problems. Journal of Experimental Psychology: General, 127(4), 355–376.

    Google Scholar 

  13. Chen, X., Mitrovic, A., & Matthews, M. (2019a). Learning from worked examples, erroneous examples and problem solving: Towards adaptive selection of learning activities. IEEE Transactions on Learning Technologies, pages 1–1.

  14. Chen, X., Mitrovic, A., & Matthews, M. (2019b). Investigating the effect of agency on learning from worked examples, erroneous examples and problem solving. International Journal of Artificial Intelligence in Education.

  15. Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145–182.

    Google Scholar 

  16. Christenson, S.L., Reschly, A.L., & Wylie, C. (2012). Handbook of research on student engagement. Springer Science & Business Media.

  17. Cooper, S., Dann, W., & Pausch, R. (2003). Teaching objects-first in introductory computer science. In: ACM SIGCSE Bulletin, vol. 35, pp. 191–195. ACM.

  18. Dann, W., Cosgrove, D., Slater, D., Culyba, D., & Cooper, S. (2012). Mediated transfer: Alice 3 to java. In: Proceedings of the 43rd ACM technical symposium on Computer Science Education, pp. 141–146. ACM.

  19. Davidovic, A., Warren, J., & Trichina, E. (2003). Learning benefits of structural example-based adaptive tutoring systems. IEEE Transactions on Education, 46(2), 241–251.

    Google Scholar 

  20. Ericson, B.J., Guzdial, M.J., & Morrison, B.B. (2015). Analysis of interactive features designed to enhance learning in an ebook. In: Proceedings of the 11th annual international conference on international computing education research, pp. 169–178 ACM.

  21. Ericson, B.J, Margulieux, L.E., & Rick, J. (2017). Solving parsons problems versus fixing and writing code. In: Proceedings of the 17th Koli calling conference on computing education research, pp. 20–29. ACM.

  22. Esteves, M., & Mendes, A. (2003). Oop-anim, a system to support learning of basic object oriented programming concepts. In: Proceedings of compsystech’2003-international conference on computer systems and technologies, Sofia. Bulgaria.

  23. Evans, C., & Gibbons, N.J. (2007). The interactivity effect in multimedia learning. Computers & Education, 49(4), 1147–1160.

    Google Scholar 

  24. Fabic, G.V.F., Mitrovic, A., & Neshatian, K. (2019). Evaluation of parsons problems with menu-based self-explanation prompts in a mobile python tutor. International Journal of Artificial Intelligence in Education. ISSN 1560-4306.

  25. Getao, K.W. (1990). An environment to support the use of program examples while learning to program in lisp. In: Proceedings of the IFIP TC13 3rd international conference on human-computer interaction, pp. 1015–1016. North-Holland Publishing Co.

  26. Hansen, S.R., Narayanan, N.H., & Schrimpsher, D. (2000). Helping learners visualize and comprehend algorithms. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 2(1).

  27. Harvey, B., & Mönig, J. (2010). Bringing “no ceiling” to scratch: Can one language serve kids and computer scientists. Proc Constructionism.

  28. Hundhausen, C.D., Douglas, S.A., & Stasko, J.T. (2002). A meta-study of algorithm visualization effectiveness. Journal of Visual Languages & Computing, 13 (3), 259–290.

    Google Scholar 

  29. Ihantola, P., & Karavirta, V. (2011). Two-Dimensional Parson’s Puzzles: The Concept, Tools, and First Observations. Journal of Information Technology Education: Innovations in Practice, 10, 1–14.

    Google Scholar 

  30. Kalyuga, S., Chandler, P., & Sweller, J. (2000). Incorporating learner experience into the design of multimedia instruction. Journal of Educational Psychology, 92(1), 126–136.

    Google Scholar 

  31. Kalyuga, S., Chandler, P., Tuovinen, J., & Sweller, J. (2001). When problem solving is superior to studying worked examples. Journal of Educational Psychology, 93(3), 579–588.

    Google Scholar 

  32. Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31.

    Google Scholar 

  33. Kay, R.H., & Knaack, L. (2009). Assessing learning, quality and engagement in learning objects: The learning object evaluation scale for students (loes-s). Educational Technology Research and Development, 57(2), 147–168.

    Google Scholar 

  34. Khandwala, K., & Guo, P.J. (2018). Codemotion: expanding the design space of learner interactions with computer programming tutorial videos. In: Proceedings of the Fifth Annual ACM Conference on Learning at Scale, pp. 57, 1–57, vol. 10.

  35. Lawrence, A.W. (1993). Empirical studies of the value of algorithm animation in algorithm understanding. PhD thesis: Georgia Institute of Technology.

    Google Scholar 

  36. Lieberman, H. (1987). An example-based environment for beginning programmers. In Artificial intelligence and education (pp. 135–151). Norwood: Ablex Publishing.

  37. Linn, MC., & Clancey, M. J. (1992). The case for case studies of programming problems. Communications of the ACM, 35(3), 121–132.

    Google Scholar 

  38. Loboda, T., Guerra, J., Hosseini, R., & Brusilovsky, P. (2014). Mastery grids: An open source social educational progress visualization. In de Freitas, S, Rensing, C, Muñoz Merino, P.J., & Ley, T (Eds.) 9th European conference on technology enhanced learning (EC-TEL 2014), vol. 8719 of lecture notes in computer science (pp. 235–248).

  39. Loboda, T.D., & Brusilovsky, P. (2010). User-adaptive explanatory program visualization: Evaluation and insights from eye movements. User Modeling and User-Adapted Interaction, 20(3), 191–226.

    Google Scholar 

  40. McLaren, BM., Adams, D., Durkin, K., Goguadze, G., Mayer, RE., Rittle-Johnson, B., Sosnovsky, S., Isotani, S., & van Velsen, M. (2012). To err is human, to explain and correct is divine: A study of interactive erroneous examples with middle school math students. In: 7th European conference on technology enhanced learning (EC-TEL 2012), vol. 7563 of lecture notes in computer science, pp. 222–235.

  41. Miller, B.N., & Ranum, D.L. (2012). Beyond pdf and epub: toward an interactive textbook. In: Proceedings of the 17th ACM annual conference on innovation and technology in computer science education, pp. 150–155. ACM.

  42. Miyadera, Y., Kurasawa, K., Nakamura, S., Yonezawa, N., & Yokoyama, S. (2007). A real-time monitoring system for programming education using a generator of program animation systems. JCP, 2(3), 12–20.

    Google Scholar 

  43. Morrison, B.B., Margulieux, L.E., Ericson, B., & Guzdial, M. (2016). Subgoals help students solve parsons problems. In: Proceedings of the 47th ACM Technical Symposium on Computing Science Education, pages 42–47 ACM.

  44. Myller, N. (2006). Automatic prediction question generation during program visualization. In: Proceedings of the 4th program visualization workshop.

  45. Najar, A.S., Mitrovic, A., & McLaren, B.M. (2016). Learning with intelligent tutors and worked examples: Selecting learning activities adaptively leads to better learning outcomes than a fixed curriculum. User Modeling and User-Adapted Interaction, 26(5), 459–491. ISSN 1573-1391.

    Google Scholar 

  46. Naps, T.L. (2005). Jhavé: Supporting algorithm visualization. IEEE Computer Graphics and Applications, 25(5), 49–55.

    Google Scholar 

  47. Naps, T.L, Eagan, J.R, & Norton, L.L. (2000). Jhavé – an environment to actively engage students in web-based algorithm visualizations. In: ACM SIGCSE bulletin, vol. 32, pp. 109–113. ACM.

  48. Naps, TL., Rößling, G., Almstrum, V., Dann, W., Fleischer, R., Hundhausen, C., Korhonen, A., Malmi, L., McNally, M., Rodger, S., & Velázquez-Iturbide, J.Á. (2002). Exploring the role of visualization and engagement in computer science education. ACM SIGCSE bulletin, 35, 131–152.

    Google Scholar 

  49. Nokes-Malach, T.J, VanLehn, K., Belenky, D.M., Lichtenstein, M., & Cox, G. (2013). Coordinating principles and examples through analogy and self-explanation. European Journal of Psychology of Education, 28(4), 1237–1263.

    Google Scholar 

  50. Nunnally, J.C. (1978). Psychometric Ttheory: 2d Ed. McGraw-Hill.

  51. Paas, F.G.W.C., & Van Merriënboer, J.J.G. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of educational psychology, 86(1), 122–133.

    Google Scholar 

  52. Park, J., Park, Y.H., Kim, J., Cha, J., Kim, S., & Alice, O.H. (2018). Elicast: embedding interactive exercises in instructional programming screencasts. In Proceedings of the 5th annual ACM conference on learning at scale, pp. 58, (Vol. 10 pp. 1–58).

  53. Parsons, D., & Haden, P. (2006). Parson’s programming puzzles: a fun and effective learning tool for first programming courses. In: Proceedings of the 8th Australasian conference on computing Education-Volume 52, pp. 157–163. Australian Computer Society Inc.

  54. Pirolli, P.L., & Anderson, J.R. (1985). The role of learning from examples in the acquisition of recursive programming skills. Canadian Journal of Psychology/Revue canadienne de psychologie, 39(2), 240–272.

    Google Scholar 

  55. Reeve, J. (2013). How students create motivationally supportive learning environments for themselves: The concept of agentic engagement. Journal of Educational Psychology, 105(3), 579–595.

    MathSciNet  Google Scholar 

  56. Reeve, J., & Tseng, C.-M. (2011). Agency as a fourth aspect of students’ engagement during learning activities. Contemporary Educational Psychology, 36(4), 257–267.

    Google Scholar 

  57. Renkl, A. (1997). Learning from worked-out examples: A study on individual differences. Cognitive science, 21(1), 1–29.

    Google Scholar 

  58. Renkl, A., & Atkinson, R. (2007). An example order for cognitive skill acquisition. Oxford University Press.

  59. Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., & et al. (2009). Scratch: programming for all. Communications of the ACM, 52(11), 60–67.

    Google Scholar 

  60. Rivers, K. (2017). Automated data-driven hint generation for learning programming. PhD thesis: Carnegie Mellon University.

    Google Scholar 

  61. Sajaniemi, J., & Kuittinen, M. (2003). Program animation based on the roles of variables. In: Proceedings of the ACM symposium on Software visualization, pp. 7–ff. ACM.

  62. Salden, R.J. C. M., Aleven, V., Schwonke, R., & Renkl, A. (2010a). The expertise reversal effect and worked examples in tutored problem solving. Instructional Science, 38(3), 289–307. ISSN 1573- 1952.

    Google Scholar 

  63. Salden, R.J.C.M., Koedinger, K.R., Renkl, A., Aleven, V., & McLaren, B.M. (2010b). Accounting for beneficial effects of worked examples in tutored problem solving. Educational Psychology Review, 22(4), 379–392.

    Google Scholar 

  64. Sears, A., & Wolfe, R. (1995). Visual analysis: Adding breadth to a computer graphics course. In: ACM SIGCSE Bulletin, vol. 27, pp. 195–198. ACM.

  65. Sharrock, R., Hamonic, E., Hiron, M., & Carlier, S. (2017). Codecast: An innovative technology to facilitate teaching and learning computer programming in a c language online course. In: Proceedings of the 4th ACM conference on learning at scale, pp. 147–148. ACM.

  66. Sirkiä, T. (2013). A javascript library for visualizing program execution. In: Proceedings of the 13th Koli calling international conference on computing education research, pp. 189–190. ACM.

  67. Skinner, E.A., Kindermann, T.A., & Furrer, C.J. (2009). A motivational perspective on engagement and disaffection: Conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educational and Psychological Measurement, 69(3), 493–525.

    MathSciNet  Google Scholar 

  68. Sorva, J., Karavirta, V., & Malmi, L. (2013). A review of generic program visualization systems for introductory programming education. ACM Transactions on Computing Education (TOCE), 13(4), 15:1–15:64.

    Google Scholar 

  69. Sweller, J., & Cooper, G.A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and instruction, 2(1), 59–89.

    Google Scholar 

  70. Sweller, J., Van Merrienboer, J.J.G., & Paas, F.G.W.C. (1998). Cognitive architecture and instructional design. Educational psychology review, 10(3), 251–296.

    Google Scholar 

  71. Trafton, J.G., & Reiser, B.J. (1993). The contributions of studying examples and solving problems to skill acquisition. In: Proceedings of the 15th annual conference of the cognitive science society, pp. 1017–1022 ACM.

  72. Walker, C.O., Greene, B.A., & Mansell, R.A. (2006). Identification with academics, intrinsic/extrinsic motivation, and self-efficacy as predictors of cognitive engagement. Learning and individual differences, 16(1), 1–12.

    Google Scholar 

  73. Ward, M., & Sweller, J. (1990). Structuring effective worked examples. Cognition and instruction, 7(1), 1–39.

    Google Scholar 

  74. Weber, G. (1996). Individual selection of examples in an intelligent learning environment. Journal of Interactive Learning Research, 7(1), 3–31.

    Google Scholar 

  75. Weber, G., & Brusilovsky, P. (2001). Elm-art: An adaptive versatile system for web-based instruction. International Journal of Artificial Intelligence in Education (IJAIED), 12, 351–384.

    Google Scholar 

  76. Weber, G., & Mollenberg, A. (1994). Elm-pe: A knowledge-based programming environment for learning lisp. In: Proceedings of ED-MEDIA 1994, pp. 557–562 ERIC.

  77. Yudelson, M., Brusilovsky, P., & Zadorozhny, V. (2007). A user modeling server for contemporary adaptive hypermedia: An evaluation of push approach to evidence propagation. In Conati, C., McCoy, K., & Paliouras, G. (Eds.) 11th International Conference on User Modeling, UM 2007, vol. 4511 of Lecture Notes in Computer Science, pp/ 27–36. Springer Verlag.

  78. Zhi, R., Price, T.W., Marwan, S., Milliken, A., Barnes, T., & Chi, M. (2019). Exploring the impact of worked examples in a novice programming environment. In: Proceedings of the 50th ACM technical symposium on computer science education, SIGCSE ’19, pp. 98–104. ACM.

Download references

Author information



Corresponding author

Correspondence to Roya Hosseini.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Hosseini, R., Akhuseyinoglu, K., Brusilovsky, P. et al. Improving Engagement in Program Construction Examples for Learning Python Programming. Int J Artif Intell Educ 30, 299–336 (2020).

Download citation


  • Introductory programming education
  • CS1
  • Python
  • Program construction
  • Worked examples
  • Classroom study