Skip to main content

Conversational agents for academically productive talk: a comparison of directed and undirected agent interventions

Abstract

Conversational agents that draw on the framework of academically productive talk (APT) have been lately shown to be effective in helping learners sustain productive forms of peer dialogue in diverse learning settings. Yet, literature suggests that more research is required on how learners respond to and benefit from such flexible agents in order to fine-tune the design of automated APT intervention modes and, thus, enhance agent pedagogical efficacy. Building on this line of research, this work explores the impact of a configurable APT agent that prompts peers to build on prior knowledge and logically connect their contributions to important domain concepts discussed in class. A total of 96 computer science students engaged in a dialogue-based activity in the context of a Human-Computer Interaction (HCI) university course. During the activity, students worked online in dyads to accomplish a learning task. The study compares three conditions: students who collaborated without any agent interference (control), students who received undirected agent interventions that addressed both peers in the dyad (U treatment), and students who received directed agent interventions addressing a particular learner instead of the dyad (D treatment). The results suggest that although both agent intervention methods can improve students’ learning outcomes and dyad in-task performance, the directed one is more effective than the undirected one in enhancing individual domain knowledge acquisition and explicit reasoning. Furthermore, findings show that the positive effect of the agent on dyad performance is mediated by the frequency of students’ contributions displaying explicit reasoning, while most students perceive agent involvement favorably.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Adamson, D., & Rosé, C. P. (2013). Academically Productive Talk: One Size Does Not Fit All. In Artificial Intelligence in Education (AIED) 2013 Workshops Proceedings (p. 51–60).

  2. Adamson, D., Ashe, C., Jang, H., Yaron, D., & Rosé, C. P, (2013). Intensification of group knowledge exchange with academically productive talk agents. In N. Rummel, M. Kapur, M. Nathan, & S. Puntambekar (Eds.), To See the World and a Grain of Sand: Learning across Levels of Space, Time, and Scale: CSCL 2013 Conference Proceedings (vol. 1, pp. 10–17).

  3. Adamson, D., Dyke, G., Jang, H., & Rosé, C. P. (2014). Towards an agile approach to adapting dynamic collaboration support to student needs. International Journal of Artificial Intelligence in Education, 24(1), 92–124.

    Article  Google Scholar 

  4. Asterhan, C. S., & Schwarz, B. B. (2016). Argumentation for learning: well-trodden paths and unexplored territories. Educational Psychologist, 51(2), 164–187.

    Article  Google Scholar 

  5. Boeije, H. (2002). A purposeful approach to the constant comparative method in the analysis of qualitative interviews. Quality and Quantity, 36(4), 391–409.

    Article  Google Scholar 

  6. Brandom, R. (1998). Making it explicit: reasoning, representing, and discursive commitment. Harvard University Press.

  7. Cafaro, A., Glas, N., & Pelachaud, C. (2016). The Effects of Interrupting Behavior on Interpersonal Attitude and Engagement in Dyadic Interactions. In J. Thangarajah, K. Tuyls, C. Jonker, & S. Marsella (Eds.), 15th International Conference on Autonomous Agents and Multiagent Systems (pp. 911–920).

  8. Chi, M. T. (2009). Active-constructive-interactive: a conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1(1), 73–105.

    Article  Google Scholar 

  9. Dillenbourg, P., & Tchounikine, P. (2007). Flexibility in macro-scripts for computer-supported collaborative learning. Journal of Computer Assisted Learning, 23(1), 1–13.

    Article  Google Scholar 

  10. Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6, 241–252.

    Article  Google Scholar 

  11. Dyke, G., Adamson, D., Howley, I., & Rosé, C. P. (2013). Enhancing scientific reasoning and discussion with conversational agents. Learning Technologies, IEEE Transactions on, 6(3), 240–247.

    Article  Google Scholar 

  12. Fischer, F., Kollar, I., Mandl, H., & Haake, J. (Eds.) (2007). Scripting computer-supported communication of knowledge. Cognitive, computational and educational perspectives. New York: Springer.

    Google Scholar 

  13. Fischer, F., Kollar, I., Stegmann, K., & Wecker, C. (2013). Toward a script theory of guidance in computer-supported collaborative learning. Educational Psychologist, 48(1), 56–66.

    Article  Google Scholar 

  14. Goodman, B. A., Linton, F. N., Gaimari, R. D., Hitzeman, J. M., Ross, H. J., & Zarrella, G. (2005). Using dialogue features to predict trouble during collaborative learning. User Modeling and User-Adapted Interaction, 15(1–2), 85–134.

    Article  Google Scholar 

  15. Gulz, A., Haake, M., Silvervarg, A., Sjödén, B., & Veletsianos, G. (2011). Building a social conversational pedagogical agent: design challenges and methodological approaches. In D. Perez-Marin & I. Pascual-Nieto (Eds.), Conversational Agents and Natural Language Interaction: Techniques and Effective Practices (pp. 128–155). IGI Global.

  16. Harrer, A., McLaren, B. M., Walker, E., Bollen, L., & Sewall, J. (2006). Creating cognitive tutors for collaborative learning: steps toward realization. User Modeling and User-Adapted Interaction, 16(3–4), 175–209.

    Article  Google Scholar 

  17. Hayes, A. F. (2013). An introduction to mediation, moderation, and conditional process analysis: a regression-based approach. New York, NY: Guilford Press.

    Google Scholar 

  18. Hmelo-Silver, C. E. (2013). Multivocality as a tool for design-based research. In D. D. Suthers, K. Lund, C. P. Rosé, C. Teplovs, & N. Law (Eds.), Productive multivocality in the analysis of group interactions (pp. 561–573). New York: Springer.

    Chapter  Google Scholar 

  19. Howley, I., Kumar, R., Mayfield, E., Dyke, G., & Rosé, C. P. (2013). Gaining insights from sociolinguistic style analysis for redesign of conversational agent based support for collaborative learning. In D. Suthers, K. Lund, C. P. Rosé, C. Teplovs, & N. Law (Eds.), Productive multivocality in the analysis of group interactions (pp. 477–494) .Springer US

    Google Scholar 

  20. Huitt, W. (2011). Bloom et al.’s taxonomy of the cognitive domain. In Educational Psychology Interactive. Valdosta, GA: Valdosta State University.

    Google Scholar 

  21. Kumar, R., & Rosé, C. P. (2011). Architecture for building conversational agents that support collaborative learning. IEEE Transactions on Learning Technologies, 4(1), 21–34.

    Article  Google Scholar 

  22. Liu, C. C., & Tsai, C. C. (2008). An analysis of peer interaction patterns as discoursed by on-line small group problem-solving activity. Computers & Education, 50(3), 627–639.

    Article  Google Scholar 

  23. Ludvigsen, S., & Mørch, A. (2010). Computer-supported collaborative learning: basic concepts, multiple perspectives, and emerging trends. In B. McGaw, P. Peterson, & E. Baker (Eds.), The international encyclopedia of education 3rd edition, volume 5 (pp. 290–296). Oxford, UK: Elsevier.

    Chapter  Google Scholar 

  24. Magnisalis, I., Demetriadis, S., & Karakostas, A. (2011). Adaptive and intelligent systems for collaborative learning support: a review of the field. IEEE Transactions on Learning Technologies, 4(1), 5–20.

    Article  Google Scholar 

  25. Michaels, S., & O’Connor, C. (2013). Conceptualizing talk moves as tools: professional development approaches for academically productive discussion. In L. B. Resnick, C. Asterhan, & S. N. Clarke (Eds.), Socializing intelligence through talk and dialogue. Washington DC: American Educational Research Association.

    Google Scholar 

  26. Michaels, S., O’Connor, C., & Resnick, L. B. (2008). Deliberative discourse idealized and realized: accountable talk in the classroom and in civic life. Studies in Philosophy and Education, 27(4), 283–297.

    Article  Google Scholar 

  27. Michaels, S., O’Connor, M. C., Hall, M. W., & Resnick L. B. (2010). Accountable Talk Sourcebook: For Classroom That Works. University of Pittsburgh Institute for Learning. Retrieved on May 1, 2016, from http://ifl.pitt.edu/index.php/download/index/ats.

  28. Noroozi, O., Teasley, S., Biemans, H. A., Weinberger, A., & Mulder, M. (2013). Facilitating learning in multidisciplinary groups with transactive CSCL scripts. International Journal of Computer-Supported Collaborative Learning, 8(2), 189–223.

    Article  Google Scholar 

  29. Oehl, M., & Pfister, H. R. (2010). E-collaborative knowledge construction in chat environments. E-Collaborative Knowledge Construction: Learning from Computer-Supported and Virtual Environments, 54–72.

  30. Papadopoulos, P. M., Demetriadis, S., & Weinberger, A. (2013). ‘Make it explicit!’: improving collaboration through increase of script coercion. Journal of Computer Assisted Learning, 29(4), 383–398.

    Article  Google Scholar 

  31. Preece, J., Sharp, H., & Rogers, Y. (2015). Interaction design: beyond human-computer interaction. John Wiley & Sons.

  32. Resnick, L. B., Michaels, S., & O’Connor, C. (2010). How (well structured) talk builds the mind. In R. Sternberg & D. Preiss (Eds.), From genes to context: new discoveries about learning from educational research and their applications (pp. 163–194). New York: Springer.

    Google Scholar 

  33. Rus, V., D’Mello, S., Hu, X., & Graesser, A. (2013). Recent advances in conversational intelligent tutoring systems. AI Magazine, 34(3), 42–54.

    Google Scholar 

  34. Sionti, M., Ai, H., Rosé, C. P., & Resnick, L. (2012). A framework for analyzing development of argumentation through classroom discussions. In N. Pinkwart & B. McLaren (Eds.), Educational Technologies for Teaching Argumentation Skills (pp. 28–55). Bentham Science Publishers.

  35. Slavin, R. E. (1992). When and why does cooperative learning increase achievement? Theoretical and empirical perspectives. In R. Hertz-Lazarowitz & N. Miller (Eds.), Interaction in cooperative groups. The theoretical anatomy of group learning (pp. 145–173). Cambridge: Cambridge University Press.

    Google Scholar 

  36. Sohmer, R., Michaels, S., O’Connor, M. C., & Resnick, L. (2009). Guided construction of knowledge in the classroom. In B. Schwarz, T. Dreyfus, & R. Hershkowitz (Eds.), Transformation of knowledge through classroom interaction (pp. 105–129). New York: Taylor and Francis.

    Google Scholar 

  37. Stahl, G. (2015). Computer-supported academically productive discourse. Socializing intelligence through academic talk and dialogue, 213–224. Retrieved on May 1, 2016 from http://gerrystahl.net/pub/lrdc2015.pdf.

  38. Stahl, G., & Rosé, C. P. (2011). Group cognition in online teams. In E. Salas & S. M. Fiore (Eds.), Theories of team cognition: cross-disciplinary perspectives. New York, NY: Taylor & Francis.

    Google Scholar 

  39. Stahl, G., Cress, U., Ludvigsen, S., & Law, N. (2014). Dialogic foundations of CSCL. International Journal of Computer-Supported Collaborative Learning, 9(2), 117.

    Article  Google Scholar 

  40. Tegos, S. (2016). Web-based conversational agents for collaborative learning support (Unpublished doctoral dissertation). Aristotle University of Thessaloniki, Thessaloniki, Greece.

  41. Tegos, S., & Demetriadis, S. (in press). Conversational agents improve peer learning through building on prior knowledge. Educational Technology & Society.

  42. Tegos, S., Demetriadis, S., & Karakostas, A. (2014). Conversational Agent to Promote Students’ Productive Talk: The Effect of Solicited vs. Unsolicited Agent Intervention. In IEEE 14th International Conference on Advanced Learning Technologies (pp. 72–76).

  43. Tegos, S., Demetriadis, S., & Karakostas, A. (2015). Promoting academically productive talk with conversational agent interventions in collaborative learning settings. Computers & Education, 87, 309–325.

    Article  Google Scholar 

  44. Vogel, F., Wecker, C., Kollar, I., & Fischer, F. (2016). Socio-Cognitive Scaffolding with Computer-Supported Collaboration Scripts: a Meta-Analysis. Educational Psychology Review, 1–35.

  45. Webb, N. M. (2009). The teacher’s role in promoting collaborative dialogue in the classroom. British Journal of Educational Psychology, 79(1), 1–28.

    Article  Google Scholar 

  46. Weinberger, A. (2011). Principles of transactive computer-supported collaboration scripts. Nordic Journal of Digital Literacy, 6(3), 189–202.

    Google Scholar 

  47. Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Computers & Education, 46(1), 71–95.

    Article  Google Scholar 

  48. Weinberger, A., Stegmann, K., & Fischer, F. (2007). Knowledge convergence in collaborative learning: concepts and assessment. Learning and Instruction, 17(4), 416–426.

    Article  Google Scholar 

  49. Wolf, M. K., Crosson, A. C., & Resnick, L. B. (2005). Accountable talk in reading comprehension instruction. Regents of the University of California.

Download references

Acknowledgments

We are appreciative of Fotini Bourotzoglou’s contribution to this work.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Stergios Tegos.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Tegos, S., Demetriadis, S., Papadopoulos, P.M. et al. Conversational agents for academically productive talk: a comparison of directed and undirected agent interventions. Intern. J. Comput.-Support. Collab. Learn 11, 417–440 (2016). https://doi.org/10.1007/s11412-016-9246-2

Download citation

Keywords

  • Conversational agent
  • Academically productive talk
  • Computer-supported collaborative learning
  • Peer dialogue