Abstract
Block-based programming environments are increasingly used to introduce computing concepts to beginners. However, novice students often struggle in these environments, given the conceptual and open-ended nature of programming tasks. To effectively support a student struggling to solve a given task, it is important to provide adaptive scaffolding that guides the student towards a solution. We introduce a scaffolding framework based on pop quizzes presented as multi-choice programming tasks. To automatically generate these pop quizzes, we propose a novel algorithm, PQuizSyn. More formally, given a reference task with a solution code and the student’s current attempt, PQuizSyn synthesizes new tasks for pop quizzes with the following features: (a) Adaptive (i.e., individualized to the student’s current attempt), (b) Comprehensible (i.e., easy to comprehend and solve), and (c) Concealing (i.e., do not reveal the solution code). Our algorithm synthesizes these tasks using techniques based on symbolic reasoning and graph-based code representations. We show that our algorithm can generate hundreds of pop quizzes for different student attempts on reference tasks from Hour of Code: Maze Challenge [11] and Karel [9]. We assess the quality of these pop quizzes through expert ratings using an evaluation rubric. Further, we have built an online platform for practicing block-based programming tasks empowered via pop quiz based feedback, and report results from an initial user study.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We refer to these multi-choice questions as “pop quizzes” as the framework could present these quizzes whenever a student needs help [7].
- 2.
Implementation of the algorithm is publicly available at https://github.com/machine-teaching-group/aied2022_pquizsyn_code.
- 3.
- 4.
When \(S^\text {quiz} \notin \textsc {SubStructs}(S^{\text {in,}\star })\), we set \(C^\text {seed}\) as a random instantiation of \(S^\text {quiz}\) – see algorithm variant PQS-OneHop in Sect. 5..
References
Ahmed, U.Z., et al.: Synthesizing tasks for block-based programming. In: NeurIPS (2020)
Ahmed, U.Z., Gulwani, S., Karkare, A.: Automatically generating problems and solutions for natural deduction. In: IJCAI (2013)
Akram, B., et al.: Automated assessment of computer science competencies from student programs with gaussian process regression. In: EDM (2020)
Alvin, C., Gulwani, S., Majumdar, R., Mukhopadhyay, S.: Synthesis of geometry proof problems. In: AAAI (2014)
Bunel, R., Hausknecht, M.J., Devlin, J., Singh, R., Kohli, P.: Leveraging grammar and reinforcement learning for neural program synthesis. In: ICLR (2018)
Cherenkova, Y., Zingaro, D., Petersen, A.: Identifying challenging CS1 concepts in a large problem dataset. In: SIGCSE (2014)
Cicirello, V.A.: On the role and effectiveness of pop quizzes in CS1. In: SIGCSE (2009)
CodeHS.com: CodeHS - Teaching Coding and CS. https://codehs.com/
CodeHS.com: Intro to Programming with Karel the Dog. https://codehs.com/info/curriculum/introkarel
Code.org: Code.org - Learn Computer Science. https://code.org/
Code.org: Hour of Code - Classic Maze Challenge. https://studio.code.org/s/hourofcode
Code.org: Hour of Code Initiative. https://hourofcode.com/
Cody, C., Maniktala, M., Lytle, N., Chi, M., Barnes, T.: The Impact of looking further ahead: a comparison of two data-driven unsolicited hint types on performance in an intelligent data-driven logic tutor. IJAIED (2021)
Cordova, L., Carver, J.C., Gershmel, N., Walia, G.: A comparison of inquiry-based conceptual feedback vs. traditional detailed feedback mechanisms in software testing education: an empirical investigation. In: SIGCSE (2021)
Crichton, W., Sampaio, G.G., Hanrahan, P.: Automating program structure classification. In: SIGCSE (2021)
Efremov, A., Ghosh, A., Singla, A.: Zero-shot learning of hint policy via reinforcement learning and program synthesis. In: EDM (2020)
Ene, A., Stirbu, C.: Automatic generation of quizzes for Java programming language. In: ECAI (2019)
Games, B.: Games for Tomorrow’s Programmers. https://blockly.games/
Grover, S.: Toward a framework for formative assessment of conceptual learning in K-12 computer science classrooms. In: SIGCSE (2021)
Gulwani, S., Polozov, O., Singh, R.: Program synthesis. Found. Trends® Program. Lang. 4(1–2), 1–119 (2017)
MacFarland, T.W., Yates, J.M.: Kruskal–Wallis H-test for oneway analysis of variance (ANOVA) by ranks. In: Introduction to Nonparametric Statistics for the Biological Sciences Using R, pp. 177–211. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30634-6_6
Marwan, S., Gao, G., Fisk, S., Price, T., Barnes, T.: Adaptive immediate feedback can improve novice programming engagement and intention to persist in computer science. In: ICER (2020)
McHugh, M.L.: The Chi-square test of independence. Biochemia Medica 23, 143–149 (2013)
Paaßen, B., Hammer, B., Price, T.W., Barnes, T., Gross, S., Pinkwart, N.: The continuous hint factory - providing hints in continuous and infinite spaces. JEDM 10, 1–35 (2018)
Pattis, R.E.: Karel the Robot: A Gentle Introduction to the Art of Programming. Wiley (1981)
Piech, C., Sahami, M., Huang, J., Guibas, L.J.: Autonomously generating hints by inferring problem solving policies. In: L@S (2015)
Polozov, O., O’Rourke, E., Smith, A.M., Zettlemoyer, L., Gulwani, S., Popovic, Z.: Personalized mathematical word problem generation. In: IJCAI (2015)
Price, T.W., Dong, Y., Lipovac, D.: iSnap: towards intelligent tutoring in novice programming environments. In: SIGCSE (2017)
Price, T.W., et al.: A comparison of the quality of data-driven programming hint generation algorithms. IJAIED 29, 368–395 (2019)
Price, T.W., Williams, J.J., Solyst, J., Marwan, S.: Engaging students with instructor solutions in online programming homework. In: CHI (2020)
Price, T.W., Zhi, R., Barnes, T.: Evaluation of a data-driven feedback algorithm for open-ended programming. In: EDM (2017)
Price, T.W., Zhi, R., Barnes, T.: Hint generation under uncertainty: the effect of hint quality on help-seeking behavior. In: AIED (2017)
Resnick, M., et al.: Scratch: programming for all. Commun. ACM 52, 60–67 (2009)
Rivers, K., Harpstead, E., Koedinger, K.R.: Learning curve analysis for programming: which concepts do students struggle with? In: ICER (2016)
Rivers, K., Koedinger, K.R.: Automating hint generation with solution space path construction. In: Trausan-Matu, S., Boyer, K.E., Crosby, M., Panourgia, K. (eds.) ITS 2014. LNCS, vol. 8474, pp. 329–339. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-07221-0_41
Rivers, K., Koedinger, K.R.: Data-driven hint generation in vast solution spaces: a self-improving Python programming tutor. IJAIED 27, 37–64 (2017)
Singh, R., Gulwani, S., Rajamani, S.: Automatically generating algebra problems. In: AAAI (2012)
Soltanpoor, R., Thevathayan, C., D’Souza, D.J.: Adaptive remediation for novice programmers through personalized prescriptive quizzes. In: ITiCSE (2018)
Wiese, E.S., Rafferty, A.N., Fox, A.: Linking code readability, structure, and comprehension among novices: it’s complicated. In: ICSE (2019)
Wiese, E.S., Rafferty, A.N., Kopta, D.M., Anderson, J.M.: Replicating novices’ struggles with coding style. In: ICPC (2019)
Wu, M., Mosse, M., Goodman, N.D., Piech, C.: Zero shot learning for code education: rubric sampling with deep learning inference. In: AAAI (2019)
Zhang, L., Li, B., Zhang, Q., Hsiao, I.: Does a distributed practice strategy for multiple choice questions help novices learn programming. iJET 15, 234 (2020)
Zhi, R., Marwan, S., Dong, Y., Lytle, N., Price, T.W., Barnes, T.: Toward data-driven example feedback for novice programming. In: EDM (2019)
Zimmerman, K., Rupakheti, C.R.: An automated framework for recommending program elements to novices (N). In: ASE (2015)
Acknowledgments
We would like to thank the reviewers for their feedback. Ahana Ghosh was supported by Microsoft Research through its PhD Scholarship Programme. Adish Singla acknowledges support by the European Research Council (ERC) under the Horizon Europe programme (ERC StG, grant agreement No. 101039090).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Ghosh, A., Tschiatschek, S., Devlin, S., Singla, A. (2022). Adaptive Scaffolding in Block-Based Programming via Synthesizing New Tasks as Pop Quizzes. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, vol 13355. Springer, Cham. https://doi.org/10.1007/978-3-031-11644-5_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-11644-5_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-11643-8
Online ISBN: 978-3-031-11644-5
eBook Packages: Computer ScienceComputer Science (R0)