This research is focused on how to support students’ acquisition of program construction skills through worked examples. Although examples have been consistently proven to be valuable for student’s learning, the learning technology for computer science education lacks program construction examples with interactive elements that could engage students. The goal of this work is to investigate the value of the “engaging” features in programming examples. We introduce PCEX, an online tool developed to present program construction examples in an engaging fashion. We also present the results of a controlled study with a between-subject design that was conducted in a large introductory Python programming class to compare PCEX with non-interactive worked examples focused on program construction. The results of our study show the positive impact of interactive program construction examples on student’s engagement, problem-solving performance, and learning.
This is a preview of subscription content, log in to check access.
Buy single article
Instant access to the full article PDF.
Price includes VAT for USA
Subscribe to journal
Immediate online access to all issues from 2019. Subscription will auto renew annually.
This is the net price. Taxes to be calculated in checkout.
We chose start date over due date as the time reference because some students submitted assignments after the due date was passed. Therefore, using the difference between the due date and the submission date would have made the interpretation of results more difficult, as some differences would be negative.
Using all students’ data showed the same pattern of results for this analysis.
Aiken, L.S, West, S.G, & Reno, R.R. (1991). Multiple regression: Testing and interpreting interactions. Sage.
Atkinson, R.K, & Renkl, A. (2007). Interactive example-based learning environments: Using interactive elements to encourage effective processing of worked examples. Educational Psychology Review, 19(3), 375–386.
Atkinson, R.K, Derry, S.J, Renkl, A., & Wortham, D. (2000). Learning from examples: Instructional principles from the worked examples research. Review of educational research, 70(2), 181– 214.
Auvinen, T., Hakulinen, L., & Malmi, L. (2018). Increasing students’ awareness of their behavior in online learning environments with visualizations and achievement badges. IEEE Transactions on Learning Technologies, 8(3), 261–273.
Bathgate, M., & Schunn, C. (2017). The psychological characteristics of experiences that influence science motivation and content knowledge. International Journal of Science Education, 39(17), 2402– 2432.
Bloom, S.B. (1956). Taxonomy of educational objectives, handbook I: The cognitive domain. New York: David McKay Co Inc.
Brna, P. (1998). Searching for examples with a programming techniques editor. Journal of Computing and Information Technology, 6(1), 13–26.
Brusilovsky, P. (1994). Explanatory visualization in an educational programming environment: connecting examples with general knowledge. In: 4th international conference on human-computer interaction, EWHCI’94, vol. 876 of Lecture Notes in Computer Science, pp. 202–212. Springer-Verlag.
Brusilovsky, P, & Yudelson, M.V. (2008). From webex to navex: Interactive access to annotated program examples. Proceedings of the IEEE, 96(6), 990–999.
Brusilovsky, P., Yudelson, M., & Hsiao, I-H. (2009). Problem solving examples as first class objects in educational digital libraries: Three obstacles to overcome. Journal of Educational Multimedia and Hypermedia, 18(3), 267–288.
Byrne, M. D, Catrambone, R., & Stasko, J.T. (1999). Evaluating animations as student aids in learning computer algorithms. Computers & education, 33(4), 253–278.
Catrambone, R. (1998). The subgoal learning model: Creating better examples so that students can solve novel problems. Journal of Experimental Psychology: General, 127(4), 355–376.
Chen, X., Mitrovic, A., & Matthews, M. (2019a). Learning from worked examples, erroneous examples and problem solving: Towards adaptive selection of learning activities. IEEE Transactions on Learning Technologies, pages 1–1.
Chen, X., Mitrovic, A., & Matthews, M. (2019b). Investigating the effect of agency on learning from worked examples, erroneous examples and problem solving. International Journal of Artificial Intelligence in Education.
Chi, M.T.H., Bassok, M., Lewis, M.W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145–182.
Christenson, S.L., Reschly, A.L., & Wylie, C. (2012). Handbook of research on student engagement. Springer Science & Business Media.
Cooper, S., Dann, W., & Pausch, R. (2003). Teaching objects-first in introductory computer science. In: ACM SIGCSE Bulletin, vol. 35, pp. 191–195. ACM.
Dann, W., Cosgrove, D., Slater, D., Culyba, D., & Cooper, S. (2012). Mediated transfer: Alice 3 to java. In: Proceedings of the 43rd ACM technical symposium on Computer Science Education, pp. 141–146. ACM.
Davidovic, A., Warren, J., & Trichina, E. (2003). Learning benefits of structural example-based adaptive tutoring systems. IEEE Transactions on Education, 46(2), 241–251.
Ericson, B.J., Guzdial, M.J., & Morrison, B.B. (2015). Analysis of interactive features designed to enhance learning in an ebook. In: Proceedings of the 11th annual international conference on international computing education research, pp. 169–178 ACM.
Ericson, B.J, Margulieux, L.E., & Rick, J. (2017). Solving parsons problems versus fixing and writing code. In: Proceedings of the 17th Koli calling conference on computing education research, pp. 20–29. ACM.
Esteves, M., & Mendes, A. (2003). Oop-anim, a system to support learning of basic object oriented programming concepts. In: Proceedings of compsystech’2003-international conference on computer systems and technologies, Sofia. Bulgaria.
Evans, C., & Gibbons, N.J. (2007). The interactivity effect in multimedia learning. Computers & Education, 49(4), 1147–1160.
Fabic, G.V.F., Mitrovic, A., & Neshatian, K. (2019). Evaluation of parsons problems with menu-based self-explanation prompts in a mobile python tutor. International Journal of Artificial Intelligence in Education. ISSN 1560-4306.
Getao, K.W. (1990). An environment to support the use of program examples while learning to program in lisp. In: Proceedings of the IFIP TC13 3rd international conference on human-computer interaction, pp. 1015–1016. North-Holland Publishing Co.
Hansen, S.R., Narayanan, N.H., & Schrimpsher, D. (2000). Helping learners visualize and comprehend algorithms. Interactive Multimedia Electronic Journal of Computer-Enhanced Learning, 2(1).
Harvey, B., & Mönig, J. (2010). Bringing “no ceiling” to scratch: Can one language serve kids and computer scientists. Proc Constructionism.
Hundhausen, C.D., Douglas, S.A., & Stasko, J.T. (2002). A meta-study of algorithm visualization effectiveness. Journal of Visual Languages & Computing, 13 (3), 259–290.
Ihantola, P., & Karavirta, V. (2011). Two-Dimensional Parson’s Puzzles: The Concept, Tools, and First Observations. Journal of Information Technology Education: Innovations in Practice, 10, 1–14.
Kalyuga, S., Chandler, P., & Sweller, J. (2000). Incorporating learner experience into the design of multimedia instruction. Journal of Educational Psychology, 92(1), 126–136.
Kalyuga, S., Chandler, P., Tuovinen, J., & Sweller, J. (2001). When problem solving is superior to studying worked examples. Journal of Educational Psychology, 93(3), 579–588.
Kalyuga, S., Ayres, P., Chandler, P., & Sweller, J. (2003). The expertise reversal effect. Educational Psychologist, 38(1), 23–31.
Kay, R.H., & Knaack, L. (2009). Assessing learning, quality and engagement in learning objects: The learning object evaluation scale for students (loes-s). Educational Technology Research and Development, 57(2), 147–168.
Khandwala, K., & Guo, P.J. (2018). Codemotion: expanding the design space of learner interactions with computer programming tutorial videos. In: Proceedings of the Fifth Annual ACM Conference on Learning at Scale, pp. 57, 1–57, vol. 10.
Lawrence, A.W. (1993). Empirical studies of the value of algorithm animation in algorithm understanding. PhD thesis: Georgia Institute of Technology.
Lieberman, H. (1987). An example-based environment for beginning programmers. In Artificial intelligence and education (pp. 135–151). Norwood: Ablex Publishing.
Linn, MC., & Clancey, M. J. (1992). The case for case studies of programming problems. Communications of the ACM, 35(3), 121–132.
Loboda, T., Guerra, J., Hosseini, R., & Brusilovsky, P. (2014). Mastery grids: An open source social educational progress visualization. In de Freitas, S, Rensing, C, Muñoz Merino, P.J., & Ley, T (Eds.) 9th European conference on technology enhanced learning (EC-TEL 2014), vol. 8719 of lecture notes in computer science (pp. 235–248).
Loboda, T.D., & Brusilovsky, P. (2010). User-adaptive explanatory program visualization: Evaluation and insights from eye movements. User Modeling and User-Adapted Interaction, 20(3), 191–226.
McLaren, BM., Adams, D., Durkin, K., Goguadze, G., Mayer, RE., Rittle-Johnson, B., Sosnovsky, S., Isotani, S., & van Velsen, M. (2012). To err is human, to explain and correct is divine: A study of interactive erroneous examples with middle school math students. In: 7th European conference on technology enhanced learning (EC-TEL 2012), vol. 7563 of lecture notes in computer science, pp. 222–235.
Miller, B.N., & Ranum, D.L. (2012). Beyond pdf and epub: toward an interactive textbook. In: Proceedings of the 17th ACM annual conference on innovation and technology in computer science education, pp. 150–155. ACM.
Miyadera, Y., Kurasawa, K., Nakamura, S., Yonezawa, N., & Yokoyama, S. (2007). A real-time monitoring system for programming education using a generator of program animation systems. JCP, 2(3), 12–20.
Morrison, B.B., Margulieux, L.E., Ericson, B., & Guzdial, M. (2016). Subgoals help students solve parsons problems. In: Proceedings of the 47th ACM Technical Symposium on Computing Science Education, pages 42–47 ACM.
Myller, N. (2006). Automatic prediction question generation during program visualization. In: Proceedings of the 4th program visualization workshop.
Najar, A.S., Mitrovic, A., & McLaren, B.M. (2016). Learning with intelligent tutors and worked examples: Selecting learning activities adaptively leads to better learning outcomes than a fixed curriculum. User Modeling and User-Adapted Interaction, 26(5), 459–491. ISSN 1573-1391.
Naps, T.L. (2005). Jhavé: Supporting algorithm visualization. IEEE Computer Graphics and Applications, 25(5), 49–55.
Naps, T.L, Eagan, J.R, & Norton, L.L. (2000). Jhavé – an environment to actively engage students in web-based algorithm visualizations. In: ACM SIGCSE bulletin, vol. 32, pp. 109–113. ACM.
Naps, TL., Rößling, G., Almstrum, V., Dann, W., Fleischer, R., Hundhausen, C., Korhonen, A., Malmi, L., McNally, M., Rodger, S., & Velázquez-Iturbide, J.Á. (2002). Exploring the role of visualization and engagement in computer science education. ACM SIGCSE bulletin, 35, 131–152.
Nokes-Malach, T.J, VanLehn, K., Belenky, D.M., Lichtenstein, M., & Cox, G. (2013). Coordinating principles and examples through analogy and self-explanation. European Journal of Psychology of Education, 28(4), 1237–1263.
Nunnally, J.C. (1978). Psychometric Ttheory: 2d Ed. McGraw-Hill.
Paas, F.G.W.C., & Van Merriënboer, J.J.G. (1994). Variability of worked examples and transfer of geometrical problem-solving skills: A cognitive-load approach. Journal of educational psychology, 86(1), 122–133.
Park, J., Park, Y.H., Kim, J., Cha, J., Kim, S., & Alice, O.H. (2018). Elicast: embedding interactive exercises in instructional programming screencasts. In Proceedings of the 5th annual ACM conference on learning at scale, pp. 58, (Vol. 10 pp. 1–58).
Parsons, D., & Haden, P. (2006). Parson’s programming puzzles: a fun and effective learning tool for first programming courses. In: Proceedings of the 8th Australasian conference on computing Education-Volume 52, pp. 157–163. Australian Computer Society Inc.
Pirolli, P.L., & Anderson, J.R. (1985). The role of learning from examples in the acquisition of recursive programming skills. Canadian Journal of Psychology/Revue canadienne de psychologie, 39(2), 240–272.
Reeve, J. (2013). How students create motivationally supportive learning environments for themselves: The concept of agentic engagement. Journal of Educational Psychology, 105(3), 579–595.
Reeve, J., & Tseng, C.-M. (2011). Agency as a fourth aspect of students’ engagement during learning activities. Contemporary Educational Psychology, 36(4), 257–267.
Renkl, A. (1997). Learning from worked-out examples: A study on individual differences. Cognitive science, 21(1), 1–29.
Renkl, A., & Atkinson, R. (2007). An example order for cognitive skill acquisition. Oxford University Press.
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., & et al. (2009). Scratch: programming for all. Communications of the ACM, 52(11), 60–67.
Rivers, K. (2017). Automated data-driven hint generation for learning programming. PhD thesis: Carnegie Mellon University.
Sajaniemi, J., & Kuittinen, M. (2003). Program animation based on the roles of variables. In: Proceedings of the ACM symposium on Software visualization, pp. 7–ff. ACM.
Salden, R.J. C. M., Aleven, V., Schwonke, R., & Renkl, A. (2010a). The expertise reversal effect and worked examples in tutored problem solving. Instructional Science, 38(3), 289–307. ISSN 1573- 1952.
Salden, R.J.C.M., Koedinger, K.R., Renkl, A., Aleven, V., & McLaren, B.M. (2010b). Accounting for beneficial effects of worked examples in tutored problem solving. Educational Psychology Review, 22(4), 379–392.
Sears, A., & Wolfe, R. (1995). Visual analysis: Adding breadth to a computer graphics course. In: ACM SIGCSE Bulletin, vol. 27, pp. 195–198. ACM.
Sharrock, R., Hamonic, E., Hiron, M., & Carlier, S. (2017). Codecast: An innovative technology to facilitate teaching and learning computer programming in a c language online course. In: Proceedings of the 4th ACM conference on learning at scale, pp. 147–148. ACM.
Skinner, E.A., Kindermann, T.A., & Furrer, C.J. (2009). A motivational perspective on engagement and disaffection: Conceptualization and assessment of children’s behavioral and emotional participation in academic activities in the classroom. Educational and Psychological Measurement, 69(3), 493–525.
Sorva, J., Karavirta, V., & Malmi, L. (2013). A review of generic program visualization systems for introductory programming education. ACM Transactions on Computing Education (TOCE), 13(4), 15:1–15:64.
Sweller, J., & Cooper, G.A. (1985). The use of worked examples as a substitute for problem solving in learning algebra. Cognition and instruction, 2(1), 59–89.
Sweller, J., Van Merrienboer, J.J.G., & Paas, F.G.W.C. (1998). Cognitive architecture and instructional design. Educational psychology review, 10(3), 251–296.
Trafton, J.G., & Reiser, B.J. (1993). The contributions of studying examples and solving problems to skill acquisition. In: Proceedings of the 15th annual conference of the cognitive science society, pp. 1017–1022 ACM.
Walker, C.O., Greene, B.A., & Mansell, R.A. (2006). Identification with academics, intrinsic/extrinsic motivation, and self-efficacy as predictors of cognitive engagement. Learning and individual differences, 16(1), 1–12.
Ward, M., & Sweller, J. (1990). Structuring effective worked examples. Cognition and instruction, 7(1), 1–39.
Weber, G. (1996). Individual selection of examples in an intelligent learning environment. Journal of Interactive Learning Research, 7(1), 3–31.
Weber, G., & Brusilovsky, P. (2001). Elm-art: An adaptive versatile system for web-based instruction. International Journal of Artificial Intelligence in Education (IJAIED), 12, 351–384.
Weber, G., & Mollenberg, A. (1994). Elm-pe: A knowledge-based programming environment for learning lisp. In: Proceedings of ED-MEDIA 1994, pp. 557–562 ERIC.
Yudelson, M., Brusilovsky, P., & Zadorozhny, V. (2007). A user modeling server for contemporary adaptive hypermedia: An evaluation of push approach to evidence propagation. In Conati, C., McCoy, K., & Paliouras, G. (Eds.) 11th International Conference on User Modeling, UM 2007, vol. 4511 of Lecture Notes in Computer Science, pp/ 27–36. Springer Verlag.
Zhi, R., Price, T.W., Marwan, S., Milliken, A., Barnes, T., & Chi, M. (2019). Exploring the impact of worked examples in a novice programming environment. In: Proceedings of the 50th ACM technical symposium on computer science education, SIGCSE ’19, pp. 98–104. ACM.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Hosseini, R., Akhuseyinoglu, K., Brusilovsky, P. et al. Improving Engagement in Program Construction Examples for Learning Python Programming. Int J Artif Intell Educ 30, 299–336 (2020). https://doi.org/10.1007/s40593-020-00197-0
- Introductory programming education
- Program construction
- Worked examples
- Classroom study