Skip to main content
SpringerLink
Log in

Using Automated Planning to Provide Feedback during Collaborative Problem-Solving

  • Article
  • Published:
International Journal of Artificial Intelligence in Education Aims and scope Submit manuscript

Abstract

Collaborative Problem-Solving Skills (CPS) have become increasingly important. Research into the development of CPS is still scarce, but there are several approaches that may be useful for its development. Specifically, providing feedback in collaborative contexts is key. In this paper, we study and develop a feedback system that uses Automated Planning techniques to promote communication among students. Our system is designed to be used in a real-world educational setting, considering the underpinning theory of when and how to give feedback. The system’s frontend is a video game, which presents tasks that can only be solved when students collaborate. In the backend, the system computes the solution to the task in a partial-order plan using an automated planning engine. While it monitors the plan and provides feedback to students. We describe an experimental study involving 42 students aged between 10 and 13, in which we explore the effectiveness of the feedback that was given. We show that the feedback allowed the students to perform better in the game, improve their communication, and develop their collaborative problem-solving skills. We also describe a novel approach to monitoring multi-agent partial-order plans, specifically designed for plans with so-called independent chains, that is more efficient than previous approaches and therefore requires fewer computational resources in the classroom. This paper contributes to the literature in two ways. First, we propose a novel monitoring algorithm for partial-order plans that is better suited to educational settings. Second, we show that feedback extracted from a plan can promote reflection about collaborative problem-solving during a multi-agent activity.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Algorithm 1:
Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 2:
Algorithm 3:

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

Code availability

The code generated during the current study is available from the corresponding author upon reasonable request.

References

  • Andrews-Todd, J., & Forsyth, C. M. (2020). Exploring social and cognitive dimensions of collaborative problem solving in an open online simulation-based task. Computers in Human Behavior, 104, 105759. https://doi.org/10.1016/j.chb.2018.10.025

  • Andrews-Todd, J., Steinberg, J., Flor, M., & Forsyth, C. (2022). Exploring automated classification approaches to advance the assessment of collaborative problem solving skills. Journal of Intelligence, 10(39), 1–24. https://doi.org/10.3390/jintelligence10030039

    Article  Google Scholar 

  • Boud, D., & Molloy, E. (2013). Feedback in higher and professional education: Understanding it and doing it well. Routledge.

    Google Scholar 

  • Bruner, J. (1985). Vygotsky: A historical and conceptual perspective. Culture, Communication, and Cognition: Vygotskian Perspectives, 21, 34.

    Google Scholar 

  • Care, E., & Griffin, P. (2014). An approach to assessment of collaborative problem solving. Research and Practice in Technology Enhanced Learning, 9(3), 367–388.

  • Chan, Y. F., KaurSidhu, G., Narasuman, S., Lee, L. F., & Yap, B. W. (2016). The relationship between collaboration in learning, quantity, and timing of feedback, and self-efficacy of students in higher education/Chan Yuen Fook…[et al.]. Social and Management Research Journal, 13(1), 60–75.

    Article  Google Scholar 

  • Chen, F., Cui, Y., & Chu, M. W. (2020). Utilizing Game Analytics to Inform and Validate Digital Game-based Assessment with Evidence-centered Game Design: A Case Study. International Journal of Artificial Intelligence in Education, 30(3), 481–503.

    Article  Google Scholar 

  • Epstein, I., & Tripodi, T. (1977). Research techniques for program planning, monitoring, and evaluation. Columbia University Press.

    Google Scholar 

  • Erhel, S., & Jamet, E. (2013). Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness. Computers & Education, 67, 156–167.

    Article  Google Scholar 

  • Freedberg, M., Glass, B., Filoteo, J. V., Hazeltine, E., & Maddox, W. T. (2017). Comparing the effects of positive and negative feedback in information-integration category learning. Memory & Cognition, 45(1), 12–25.

    Article  Google Scholar 

  • Fritz, C., & McIlraith, S. A. (2007, May). Monitoring plan optimality during execution: theory and implementation. In: The 18th International Workshop on Principles of Diagnosis (DX 2007), Nashville, TN, USA (pp. 298–305).

  • Fyfe, E. R., & Rittle-Johnson, B. (2016). The benefits of computer-generated feedback for mathematics problem-solving. Journal of experimental child psychology, 147, 140–151.

    Article  Google Scholar 

  • Ghallab, M., Nau, D., & Traverso, P. (2004). Automated Planning: theory and practice. Elsevier.

    Google Scholar 

  • Gilbert, S. B., Slavina, A., Dorneich, M. C., Sinatra, A. M., Bonner, D., Johnston, J., ... & Winer, E. (2018). Creating a team tutor using GIFT. International Journal of Artificial Intelligence in Education, 28(2), 286–313. https://doi.org/10.1007/s40593-017-0151-2

  • Goldin, I., Narciss, S., Foltz, P., & Bauer, M. (2017). New directions in formative feedback in interactive learning environments. International Journal of Artificial Intelligence in Education, 27(3), 385–392.

    Article  Google Scholar 

  • Graesser, A. C., Fiore, S. M., Greiff, S., Andrews-Todd, J., Foltz, P. W., & Hesse, F. W. (2018). Advancing the Science of Collaborative Problem Solving. The Public Interest, 19(2), 59–92.

    Article  Google Scholar 

  • Graesser, A. C., Greiff, S., Stadler, M., & Shubeck, K. T. (2020). Collaboration in the 21st century:: The theory, assessment, and teaching of collaborative problem solving.

  • Griffin, P., Care, E., & McGaw, B. (2012). The changing role of education and schools. Assessment and teaching of 21st-century skills (pp. 1–15). Springer Netherlands.

    Chapter  Google Scholar 

  • Grover, S., Chakraborti, T., & Kambhampati, S. (2018). What can Automated Planning do for Intelligent Tutoring Systems? ICAPS SPARK.

  • Hattie, J., & Gan, M. (2011). Instruction based on feedback. In: Handbook of research on learning and instruction (pp. 263–285). Routledge.

  • Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112.

    Article  Google Scholar 

  • Helmert, M. (2006). The fast downward planning system. Journal of Artificial Intelligence Research, 26, 191–246.

    Article  Google Scholar 

  • Herborn, K., Stadler, M., Mustafić, M., & Greiff, S. (2020). The assessment of collaborative problem-solving in PISA 2015: Can computer agents replace humans? Computers in Human Behavior, 104, 105624.

    Article  Google Scholar 

  • Hewett, K. J., Zeng, G., & Pletcher, B. C. (2020). The acquisition of 21st-century skills through video games: Minecraft design process models and their web of class roles. Simulation & Gaming, 51(3), 336–364.

    Article  Google Scholar 

  • Law, V., & Chen, C.-H. (2016). Promoting science learning in game-based learning with question prompts and feedback. Computers & Education, 103, 134–143. https://doi.org/10.1016/j.compedu.2016.10.005

  • McAlister, A. M., Lee, D. M., Ehlert, K. M., Kajfez, R. L., Faber, C. J., & Kennedy, M. S. (2017, June). Qualitative coding: An approach to assess inter-rater reliability. In: 2017 ASEE annual conference & exposition.

  • Muise, C., McIlraith, S. A., & Beck, J. C. (2011). Monitoring the execution of partial-order plans via regression. In IJCAI Proceedings-International Joint Conference on Artificial Intelligence (Vol. 22, No. 3, p. 1975).

  • Nadler, D. A. (1979). The effects of feedback on task group behavior: A review of the experimental research. Organizational Behavior and Human Performance, 23(3), 309–338. https://doi.org/10.1016/0030-5073(79)90001-1

  • Nadolny, L., Valai, A., Cherrez, N. J., Elrick, D., Lovett, A., & Nowatzke, M. (2020). Examining the characteristics of game-based learning: A content analysis and design framework. Computers & Education, 156, 103936. https://doi.org/10.1016/j.compedu.2020.103936

  • Nicol, D., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: A peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102–122.

    Article  Google Scholar 

  • Plass, J. L., Homer, B. D., & Kinzer, C. K. (2015). Foundations of game-based learning. Educational Psychologist, 50(4), 258–283.

    Article  Google Scholar 

  • Pea, R. D. (2004). The social and technological dimensions of scaffolding and related theoretical concepts for learning, education, and human activity. Journal of the Learning Sciences, 13(3), 423–451. https://doi.org/10.1207/s15327809jls1303_6

  • O’Donovan, B. M., den Outer, B., Price, M., & Lloyd, A. (2021). What makes good feedback good? Studies in Higher Education, 46(2), 318–329. https://doi.org/10.1080/03075079.2019.1630812

  • OECD. (2017). Pisa 2015 Collaborative Problem Solving Framework. Retrieved July 20, 2020, from https://www.oecd.org/pisa/pisaproducts/DraftPISA2015CollaborativeProblemSolvingFramework.pdf

  • OECD. (2018). PISA 2015 Results (volume V): Collaborative problem solving. PISA, OECD Publishing. https://doi.org/10.1787/9789264285521-en

  • Qian, M., & Clark, K. R. (2016). Game-based Learning and 21st-century skills: A review of recent research. Computers in Human Behavior, 63, 50–58.

    Article  Google Scholar 

  • Rojas, M., Nussbaum, M., Guerrero, O., Chiuminatto, P., Greiff, S., Del Rio, R., & Alvares, D. (2022). Integrating a collaboration script and group awareness to support group regulation and emotions towards collaborative problem solving. International Journal of Computer-Supported Collaborative Learning, 17(1), 135–168. https://doi.org/10.1007/s11412-022-09362-0

    Article  Google Scholar 

  • Rosen, Y., Wolf, I., & Stoeffler, K. (2020). Fostering collaborative problem-solving skills in science: The Animalia project. Computers in Human Behavior, 104, 105922.

    Article  Google Scholar 

  • Scoular, C., & Care, E. (2020). Monitoring patterns of social and cognitive student behaviors in online collaborative problem-solving assessments. Computers in Human Behavior, 104, 105874.

    Article  Google Scholar 

  • Scoular, C., Care,  E., & Awwal, N. (2017). An approach to scoring collaboration in online game environments. The Electronic Journal of e-Learning, 15(Issue 4), 335–342. Available online at www.ejel.org

  • Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795

  • Shute, V. J., & Becker, B. J. (2010). Innovative assessment for the 21st century. Springer.

    Book  Google Scholar 

  • Stadler, M., Herborn, K., Mustafić, M., & Greiff, S. (2020). The assessment of collaborative problem solving in PISA 2015: An investigation of the validity of the PISA 2015 CPS tasks. Computers & Education, 157, 103964. https://doi.org/10.1016/j.compedu.2020.103964

  • Sun, C., Shute, V. J., Stewart, A., Yonehiro, J., Duran, N., & D’Mello, S. (2020). Towards a generalized competency model of collaborative problem-solving. Computers & Education, 143, 103672.

    Article  Google Scholar 

  • Szewkis, E., Nussbaum, M., Rosen, T., Abalos, J., Denardin, F., Caballero, D., Tagle, A., & Alcoholado, C. (2011). Collaboration within large groups in the classroom. International Journal of Computer-Supported Collaborative Learning, 6(4), 561–575. https://doi.org/10.1007/s11412-011-9123-y

    Article  Google Scholar 

  • Taub, M., Sawyer, R., Lester, J., & Azevedo, R. (2020). The impact of contextualized emotions on self-regulated learning and scientific reasoning during learning with a game-based learning environment. International Journal of Artificial Intelligence in Education, 30(1), 97–120.

    Article  Google Scholar 

  • von Davier, A. A., Hao, J., Liu, L., & Kyllonen, P. (2017). Interdisciplinary research agenda in support of assessment of collaborative problem solving: lessons learned from developing a Collaborative Science Assessment Prototype. Computers in Human Behavior, 76, 631–640. https://doi.org/10.1016/j.chb.2017.04.059

  • Weld, D. S. (1994). An introduction to least commitment planning. AI Magazine, 15(4), 27.

    Google Scholar 

  • Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89–100. https://doi.org/10.1111/j.1469-7610.1976.tb00381.x

    Article  Google Scholar 

  • Wooten, J. O., & Ulrich, K. T. (2017). Idea generation and the role of feedback: Evidence from field experiments with innovation tournaments. Production and Operations Management, 26(1), 80–99.

    Article  Google Scholar 

  • Zhu, M., Liu, O. L., & Lee, H. S. (2020). The effect of automated feedback on revision behavior and learning gains in formative assessment of scientific argument writing. Computers & Education, 143, 103668.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by FONDECYT/CONICYT [FONDECYT 1180024].

Funding

This work was supported by 1180024.

Author information

Authors and Affiliations

  1. School of Engineering, Pontificia Universidad Católica de Chile, Av. Vicuña Mackenna 4860, Macul, Santiago, Chile

    Matias Rojas, Cristian Sáez, Jorge Baier, Miguel Nussbaum, Orlando Guerrero & María Fernanda Rodríguez

  2. Facultad de Ingeniería y Ciencias, Escuela de Informática y Telecomunicaciones, Universidad Diego Portales, Santiago, Chile

    Matias Rojas

Authors
  1. Matias Rojas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cristian Sáez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jorge Baier
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Miguel Nussbaum
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Orlando Guerrero
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. María Fernanda Rodríguez
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Matias Rojas.

Ethics declarations

Ethics approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. The ethical committee of xx approved this project under number xx.

Consent to participate

Informed consent was obtained from all individual participants included in the study and their respective legal guardians.

Consent for publication

The participant and their legal guardians have consented to the submission of the case report to the journal.

Conflicts of interest/Competing interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

ESM 1

(DOCX 815 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rojas, M., Sáez, C., Baier, J. et al. Using Automated Planning to Provide Feedback during Collaborative Problem-Solving. Int J Artif Intell Educ 33, 1057–1091 (2023). https://doi.org/10.1007/s40593-022-00321-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40593-022-00321-2

Keywords

Search

Navigation