Shifting the Load: a Peer Dialogue Agent that Encourages its Human Collaborator to Contribute More to Problem Solving

  • Cynthia Howard
  • Pamela Jordan
  • Barbara Di Eugenio
  • Sandra Katz


Despite a growing need for educational tools that support students at the earliest phases of undergraduate Computer Science (CS) curricula, relatively few such tools exist–the majority being Intelligent Tutoring Systems. Since peer interactions more readily give rise to challenges and negotiations, another way in which students can become more interactive during problem solving, we created an artificial peer collaborator to determine its value for aiding CS students. Central to its development was the notion that it should monitor the student’s collaborative behavior and attempt to guide him/her towards more productive behavior. In prior work, we found that initiative shifts correlate with both Knowledge Co-Construction (KCC) and learning and are potentially easier to model as an indicator of productive collaboration in instructional software. In this paper, we describe a unique peer dialogue agent that we created to test the effects of tracking and reacting to initiative shifts. While our study did not find differences in learning gains when comparing agents that do and do not track and react to initiative shifts, we did find that students do learn when interacting with the agent and that attempting to influence initiative taking did make a difference. This suggests that by tracking initiative shifts, the agent was able to detect times when the student had been letting the agent do most of the “deep thinking” and that the agent’s tactics for encouraging the student to begin taking the initiative again were helpful.


Peer agent Collaborative problem solving Collaborative dialogue Computer science education 



We thank our anonymous reviewers for their helpful, detailed feedback and Patricia Albacete for advice on analyzing the results. For financial support we gratefully acknowledge the National Science Foundation (ALT-0536968 and ALT-0536959) and the Office of Naval Research (N000140010640).


  1. AA. VV. (2001). Computing Curricula 2001 – Computer Science. Association for Computing Machinery, and IEEE Computer Society. Report of the Joint Task Force.Google Scholar
  2. AA. VV. (2007). Subject benchmark statement: computing. The quality assurance agency for higher education, United Kingdom.Google Scholar
  3. AA. VV. (2013). Computer science curriculum 2013. Association for Computing Machinery, and IEEE Computer Society. Report of the Joint Task Force.Google Scholar
  4. AA. VV. (2014). Occupational outlook handbook. Technical report, US Department of Labor.
  5. Albacete, P., Jordan, P.W., & Katz, S. (2015). Is a dialogue-based tutoring system that emulates helpful co-constructed relations during human tutoring effective?. In AIED 2015, the 17th international conference on artificial intelligence in education.Google Scholar
  6. Alzoubi, O., Fossati, D., Di Eugenio, B., & Green, N. (2014). ChiQat-Tutor: an integrated environment for learning recursion. In Proceedings of the 2nd workshop on AI-supported education for computer science (AIEDCS) (at ITS 2014), Honolulu, HI.Google Scholar
  7. Anderson, J.R. (1976). Language, memory, and thought. Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
  8. Austin, J.L. (1962). How to do things with words. Oxford: Oxford University Press.Google Scholar
  9. Barker, L.J., & Garvin-Doxas, K. (2004). Making visible the behaviors that influence learning environment: a qualitative exploration of computer science classrooms. Computer Science Education, 14(2), 119–145.CrossRefGoogle Scholar
  10. Barron, B. (2003). When smart groups fail. Journal of the Learning Sciences, 12(3), 307–359.CrossRefGoogle Scholar
  11. Beaubouef, T., & Mason, J. (2005). Why the high attrition rate for computer science students: some thoughts and observations. SIGCSE Bull., 37(2), 103–106.CrossRefGoogle Scholar
  12. Ben-Ari, M. (1998). Constructivism in computer science education. SIGCSE Bull., 30(1), 257–261.CrossRefGoogle Scholar
  13. Beyer, S. (2014). Why are women underrepresented in computer science? Gender differences in stereotypes, self-efficacy, values, and interests and predictors of future CS course-taking and grades. Computer Science Education, 24(2–3), 153–192.CrossRefGoogle Scholar
  14. Bhatt, K., Evens, M., & Argamon, S. (2004). Hedged responses and expressions of affect in human/human and human computer tutorial interactions. In Proceedings of the 26th annual meeting of the cognitive science society.Google Scholar
  15. Blackwell, L.S., Trzesniewski, K.H., & Dweck, C.S. (2007). Implicit theories of intelligence predict achievement across an adolescent transition: a longitudinal study and an intervention. Child Development, 78(1), 246–263.CrossRefGoogle Scholar
  16. Boyer, K., Phillips, R., Ingram, A., Ha, E., Wallis, M., Vouk, M., & Lester, J. (2011). Investigating the relationship between dialogue structure and tutoring effectiveness: a hidden markov modeling approach. International Journal of Artificial Intelligence in Education, 21(1), 65–81.Google Scholar
  17. Brienza, V. (2012). the 10 best jobs of 2012.
  18. Cai, Z., Feng, S., Baer, W., & Graesser, A. (2014). Instructional strategies in trialogue-based intelligent tutoring systems. In R.A. Sottilare, A. Graesser, X. Hu, & B. Goldberg (Eds.), Design recommendations for intelligent tutoring systems: volume 2 - instructional management. U.S. Army Research Laboratory (pp. 225–236).Google Scholar
  19. Chan, C. (2001). Peer collaboration and discourse patterns in learning from incompatible information. Instructional Science, 29(6), 443–479.CrossRefGoogle Scholar
  20. Chi, M.T.H. (2009). Active-constructive-interactive: a conceptual framework for differentiating learning activities. Topics in Cognitive Science, 1, 73–105.CrossRefGoogle Scholar
  21. Chu-Carroll, J., & Brown, M.K. (1998). An evidential model for tracking initiative in collaborative dialogue interactions. User Modeling and User-Adapted Interaction, 8(3–4), 215–253.CrossRefGoogle Scholar
  22. Cleary, J.G., & Trigg, L.E. (1995). K : an instance-based learner using an entropic distance measure. In Proceedings of the 12th international conference on machine learning (pp. 108–114).Google Scholar
  23. Cohen, W.W. (1995). Fast effective rule induction. In Proceedings of the 12th international conference on machine learning (pp. 115–123).Google Scholar
  24. Constantino-González, M., Suthers, D., & Escamilla De los Santos, J (2003). Coaching web-based collaborative learning based on problem solution differences and participation. International Journal of Artificial Intelligence in Education, 13(2–4), 263–299.Google Scholar
  25. Corbett, A.T., & Anderson, J.R. (1990). The effect of feedback control on learning to program with the LISP tutor. In Proceedings of the 12th annual conference of the cognitive science society (pp. 796–803).Google Scholar
  26. Core, M.G., Moore, J.D., & Zinn, C. (2003). The role of initiative in tutorial dialogue. In EACL ’03: Proceedings of the 10th Conference of the European Chapter of the Association for Computational Linguistics (pp. 67–74). Morristown, NJ: Association for Computational Linguistics.Google Scholar
  27. Damşa, C.I. (2014). The multi-layered nature of small-group learning: productive interactions in object-oriented collaboration. International Journal of Computer-Supported Collaborative Learning (pp. 1–35).Google Scholar
  28. DeClue, T., Kimball, J., Lu, B., & Cain, J. (2011). Five focused strategies for increasing retention in computer science 1. Journal of Computing Sciences in Colleges, 26(5), 252–258.Google Scholar
  29. Di Eugenio, B., Fossati, D., Ohlsson, S., & Cosejo, D. (2009). Towards explaining effective tutorial dialogues. In Proceedings of CogSci 2009, the annual meeting of the cognitive science society. Amsterdam, The Netherlands.Google Scholar
  30. Dzikovska, M., Steinhauser, N., Farrow, E., Moore, J., & Campbell, G. (2014). BEETLE II: deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics. International Journal of Artificial Intelligence in Education, 24(3), 284–332.CrossRefGoogle Scholar
  31. Ezen-Can, A., & Boyer, K. (2013). In-context evaluation of unsupervised dialogue act models for tutorial dialogue. In Proceedings of the SIGDIAL 2013 conference (pp. 324–328). Metz: Association for Computational Linguistics.Google Scholar
  32. Ezen-Can, A., & Boyer, K.E. (2015). A tutorial dialogue system for real-time evaluation of unsupervised dialogue act classifiers: exploring system outcomes. In Proceedings of the international conference on artificial intelligence in education.Google Scholar
  33. Falkner, N., & Falkner, K. (2012). A fast measure for identifying at-risk students in computer science. In Proceedings of the 9th annual international conference on international computing education research(pp. 55–62). ACM.Google Scholar
  34. Fisher, A., Margolis, J., & Miller, F. (1997). Undergraduate women in computer science: experience, motivation and culture. ACM SIGCSE Bulletin, 29(1), 106–110.CrossRefGoogle Scholar
  35. Fossati, D., Di Eugenio, B., Brown, C., Ohlsson, S., Cosejo, D., & Chen, L. (2009). Supporting Computer Science curriculum: exploring and learning linked lists with iList. IEEE Transactions on Learning Technologies, Special Issue on Real-World Applications of Intelligent Tutoring Systems, 2(2), 107–120.Google Scholar
  36. Fossati, D., Di Eugenio, B., Ohlsson, S., Brown, C., & Chen, L. (2015). Data driven automatic feedback generation in the iList intelligent tutoring system. Technology, Instruction, Cognition and Learning, 10(1), 5–26.Google Scholar
  37. Fuller, U., Johnson, C.G., Ahoniemi, T., Cukierman, D., Hernán-Losada, I., Jackova, J., Lahtinen, E., Lewis, T.L., Thompson, D.M., Riedesel, C., & Thompson, E. (2007). Developing a computer science-specific learning taxonomy. SIGCSE Bull., 39(4), 152–170.CrossRefGoogle Scholar
  38. Graesser, A.C., Lu, S., Jackson, G.T., Mitchell, H.H., Ventura, M., Olney, A., & Louwerse, M.M. (2004). AutoTutor: a tutor with dialogue in natural language. Behavior Research Methods, Instruments, & Computers, 36(13), 180–192.CrossRefGoogle Scholar
  39. Guinn, C.I. (1998). An analysis of initiative selection in collaborative task-oriented discourse. User Modeling and User-Adapted Interaction, 8(3–4), 255–314.CrossRefGoogle Scholar
  40. Gürer, D., & Camp, T. (2002). An acm-w literature review on women in computing. ACM SIGCSE Bulletin, 34(2), 121–127.CrossRefGoogle Scholar
  41. Harsley, R. (2015). Learning together: expanding the one-to-one its model for computer science education. In Doctoral consortium at the 11th international conference on computing education research. Omaha, NE.Google Scholar
  42. Hausmann, R.G., Chi, M.T., & Roy, M. (2004). Learning from collaborative problem solving: an analysis of three hypothesized mechanisms. In K. Forbus, D. Gentner, & T. Regier (Eds.), Proceedings of the 26th annual conference of the cognitive science society. Mahwah, NJ (pp. 547–552).Google Scholar
  43. Heeman, P.A., Yang, F., & Strayer, S.E. (2003). Control in task-oriented dialogues. In EUROSPEECH-2003 (pp. 209–212).Google Scholar
  44. Howard, C., Di Eugenio, B., Jordan, P., & Katz, S. (2015). Using initiative to operationalize knowledge co-construction during collaborative problem solving. Cognitive Science. To appear.Google Scholar
  45. Hundhausen, C.D., Douglas, S.A., & Starko, J.T. (2002). A meta-study of algorithm visualization effectiveness. Journal of Visual Languages and Computing, 13(3), 259–290.CrossRefGoogle Scholar
  46. Hunt, J.M. (1961). Intelligence and experience. Ronald.Google Scholar
  47. Jordan, P. (2007). Topic initiative in a simulated peer dialogue agent. In AIED 2007, the 13th international conference on artificial intelligence in education (pp. 581–583).Google Scholar
  48. Jordan, P.W., & Di Eugenio, B. (1997). Control and initiative in collaborative problem solving dialogues. In Working notes of the AAAI spring symposium on computational models for mixed initiative(pp. 81–84). Menlo Park, CA.Google Scholar
  49. Jordan, P.W., Hall, B., Ringenberg, M.A., Cue, Y., & Rosé, C.P. (2007). Tools for authoring a dialogue agent that participates in learning studies. In Artificial intelligence in education, AIED 2007 (pp. 43–50).Google Scholar
  50. Karlsson, F. (2007). Constraints on multiple center-embedding of clauses. Journal of Linguistics, 43, 365–392.CrossRefGoogle Scholar
  51. Katz, S. (2006). Gendered attrition at the undergraduate level. In E. Trauth (Ed.), Encyclopedia of gender and information technology (pp. 714–720). Hershey: Idea Group Publishing.CrossRefGoogle Scholar
  52. Katz, S., Allbritton, D., Aronis, J., Wilson, C., & Soffa, M.L. (2006). Gender, achievement, and persistence in an undergraduate computer science program. SIGMIS Database, 37(4), 42–57.CrossRefGoogle Scholar
  53. Katz, S., Aronis, J., Allbritton, D., Wilson, C., & Soffa, M. (2003). Gender and race in predicting achievement in computer science. IEEE Technology and Society Magazine, 22(3), 20–27.CrossRefGoogle Scholar
  54. Kersey (Howard), C. (2009). Knowledge co-construction and initiative in peer learning interactions. PhD thesis, University of Illinois at Chicago.Google Scholar
  55. Kersey (Howard), C., Di Eugenio, B., Jordan, P., & Katz, S. (2010). KSC-PaL: a peer learning agent. In ITS 2010, the 10th International conference on intelligent tutoring systems (pp. 72–81). Pittsburgh: Springer.Google Scholar
  56. Kersey (Howard), C., Di Eugenio, B., Jordan, P., & Katz, S. (2009). Knowledge co-construction and initiative in peer learning interactions. In AIED 2009, The 14th International Conference on Artificial Intelligence in Education. Brighton, UK.Google Scholar
  57. Kersey (Howard), C., Di Eugenio, B., Jordan, P.W., & Katz, S. (2008). Modeling knowledge co-construction for peer learning interactions. In ITS 2008, the 9th international conference on intelligent tutoring systems, student research workshop. Montreal, Canada.Google Scholar
  58. Kozma, R. (2003). The material features of multiple representations and their cognitive and social affordances for science understanding. Learning and Instruction, 13, 205–226.MathSciNetCrossRefGoogle Scholar
  59. Lane, H.C., & VanLehn, K. (2003). Coached program planning: dialogue-based support for novice program design. SIGCSE Bull., 35(1), 148–152.CrossRefGoogle Scholar
  60. Latham, A., Crockett, K., & McLean, D. (2014). An adaptation algorithm for an intelligent natural language tutoring system. Computers & Education, 71, 97–110.CrossRefGoogle Scholar
  61. Lehman, B., D’Mello, S., & Graesser, A. (2012). Confusion and complex learning during interactions with computer learning environments. The Internet and Higher Education, 15(3), 184–194.CrossRefGoogle Scholar
  62. Li, Z., Plaue, C., & Kraemer, E. (2013). A spirit of camaraderie: the impact of pair programming on retention. In IEEE 26th conference on software engineering education and training (CSEE&T) (pp. 209–218). IEEE.Google Scholar
  63. Lister, R., & Leaney, J. (2003). Introductory programming, criterion-referencing, and bloom. In SIGCSE’03: proceedings of the 34th SIGCSE technical symposium on computer science education (pp. 143–147). New York: ACM Press.CrossRefGoogle Scholar
  64. McCauley, R., Fitzgerald, S., Lewandowski, G., Murphy, L., Simon, B., Thomas, L., & Zander, C. (2008). Debugging: a review of the literature from an educational perspective. Computer Science Education, 18(2), 67–92.CrossRefGoogle Scholar
  65. Mead, J., Gray, S., Hamer, J., James, R., Sorva, J., Clair, C.S., & Thomas, L. (2006). A cognitive approach to identifying measurable milestones for programming skill acquisition. In ITiCSE-WGR ’06: working group reports on innovation and technology in computer science education (pp. 182–194). New York: ACM.CrossRefGoogle Scholar
  66. Meltzer, D.E. (2005). Relation between students’ problem-solving performance and representational format. American Journal of Physics, 73(5), 463–478.CrossRefGoogle Scholar
  67. Miller, G.A., & Chomsky, N. (1963). Finitary models of language users. In R.D. Luce, R.R. Bush, & E. Galanter (Eds.), Handbook of mathematical psychology, (Vol. II pp. 419–491). New York: Wiley.Google Scholar
  68. Mitrović, A., Suraweera, P., Martin, B., & Weerasinghe, A. (2004). DB-Suite: experiences with three intelligent, web-based database tutors. Journal of Interactive Learning Research, 15(4), 409–433.Google Scholar
  69. Monge, A.E., Fadjo, C.L., Quinn, B.A., & Barker, L.J. (2015). EngageCSEdu: engaging and retaining CS1 and CS2 students. ACM Inroads, 6(1), 6–11.CrossRefGoogle Scholar
  70. Mullins, D., Deiglmayr, A., & Spada, H. (2013). Motivation and emotion in shaping knowledge co-construction. In M. Baker, S. Jarvela, & J. Andriessen (Eds.) Affective learning together: social and emotional dimensions of collaborative learning (pp. 139–161). Routledge.Google Scholar
  71. Nouri, E., & Traum, D. (2014). Initiative taking in negotiation. In 15th annual meeting of the ACL Special interest group on discourse and dialogue (pp. 186–193). Philadelphia.Google Scholar
  72. Pirolli, P., & Anderson, J.R. (1985). The role of learning from examples in the acquisition of recursive programming skills. Canadian Journal of Psychology, 39(2), 240–272.CrossRefGoogle Scholar
  73. Pirolli, P., & Recker, M. (1994). Learning strategies and transfer in the domain of programming. Cognition and Instruction, 12(3), 235–275.CrossRefGoogle Scholar
  74. Porter, L., Guzdial, M., McDowell, C., & Simon, B. (2013). Success in introductory programming: what works? Communications of the ACM, 56(8), 34–36.CrossRefGoogle Scholar
  75. Reese, D.S., Lee, S., Jankun-Kelly, T., & Henderson, L. (2014). Broadening participation in computing through curricular changes. In ASEE southeast section conference.Google Scholar
  76. Renumol, V., Janakiram, D., & Jayaprakash, S. (2010). Identification of cognitive processes of effective and ineffective students during computer programming. Transactions on Computing Education, 10(3), 1–21.CrossRefGoogle Scholar
  77. Resnick, L. (1989). Knowing, learning, and instruction: essays in honor of rober glaser. Hillsdale, NJ: Erlbaum.Google Scholar
  78. Scime, A. (2008). Globalized computing education: Europe and the United States. Computer Science Education, 18(1), 43–64.CrossRefGoogle Scholar
  79. Scott, T. (2003). Bloom’s taxonomy applied to testing in computer science. In Proceedings of the 12 annual CCSC rocky mountain conference.Google Scholar
  80. Searle, J.R. (1965). What is a speech act. In M. Black (Ed.), Philosophy in America. Reprinted in Pragmatics. A Reader, Steven Davis editor, Oxford University Press, 1991 (pp. 615–628). Ithaca: Cornell University Press.Google Scholar
  81. Searle, J.R. (1975). Indirect speech acts. In P. Cole, & J. Morgan (Eds.), Syntax and semantics 3. Speech acts. Reprinted in Pragmatics. A Reader, Steven Davis editor, Oxford University Press, 1991. New York: Academic.Google Scholar
  82. Shah, F., Evens, M.W., & Michael, J.A. (2002). Classifying student initiatives and tutor responses in human keyboard-to-keyboard tutoring sessions. Discourse Processes, 33(1), 23–52.CrossRefGoogle Scholar
  83. Soh, L.-K. (2006). Incorporating an intelligent tutoring system into CS1. In SIGCSE’06 (pp. 486–490). Houston: Association for Computing Machinery.Google Scholar
  84. Soller, A. (2004). Computational modeling and analysis of knowledge sharing in collaborative distance learning. User Modeling and User-Adapted Interaction, 14(4), 351–381.CrossRefGoogle Scholar
  85. Soloway, E., & Spohrer, J.C. (1988). Studying the novice programmer. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  86. Walker, M., & Whittaker, S. (1990). Mixed initiative in dialogue: an investigation into discourse segmentation. In Proceedings of the 28th annual meeting of the association for computational linguistics (pp. 70–78). Morristown, NJ: Association for Computational Linguistics.CrossRefGoogle Scholar
  87. Wang, D., Han, H., Zhan, Z., Xu, J., Liu, Q., & Ren, G. (2015). A problem solving oriented intelligent tutoring system to improve students’ acquisition of basic computer skills. Computers & Education, 81, 102–112.CrossRefGoogle Scholar
  88. Wilson, B.C. (2002). A study of factors promoting success in Computer Science including gender differences. Computer Science Education, 12(1–2), 141–164.CrossRefGoogle Scholar
  89. Wing, J. (2006). Computational thinking. Communications of the ACM, 49 (3), 33–35.CrossRefGoogle Scholar
  90. Winslow, L.E. (1996). Programming pedagogy—a psychological overview. SIGCSE Bull., 28(3), 17–22.CrossRefGoogle Scholar
  91. Xu, S., & Rajlich, V. (2004). Cognitive process during program debugging. In Proceedings of the 3rd IEEE international conference on cognitive informatics, ICCI ’04 (pp. 176–182). IEEE Computer Society.Google Scholar
  92. Yang, F., & Heeman, P. (2010). Initiative conflicts in task-oriented dialogue. Computer Speech & Language, 24(2), 175–189.CrossRefGoogle Scholar
  93. Yoon, B., & Garcia, O. (1998). Cognitive activities and support in debugging. In Proceedings, 4th annual symposium on human interaction with complex systems (pp. 160–169). IEEE.Google Scholar
  94. Zweben, S., & Bizot, B. (2015). Relentless growth in undergraduate CS enrollment; doctoral degree production remains strong, but no new record (2013–2014 taulbee survey). Computing Research News, 27(5).Google Scholar

Copyright information

© International Artificial Intelligence in Education Society 2015

Authors and Affiliations

  • Cynthia Howard
    • 1
  • Pamela Jordan
    • 2
  • Barbara Di Eugenio
    • 3
  • Sandra Katz
    • 2
  1. 1.Computer and Mathematical Sciences DepartmentLewis UniversityRomeovilleUSA
  2. 2.Learning Research and Development CenterUniversity of PittsburghPittsburghUSA
  3. 3.Department of Computer ScienceUniversity of Illinois at ChicagoChicagoUSA

Personalised recommendations