Using Dialogue Features to Predict Trouble During Collaborative Learning

  • Bradley A. Goodman
  • Frank N. Linton
  • Robert D. Gaimari
  • Janet M. Hitzeman
  • Helen J. Ross
  • Guido Zarrella


A web-based, collaborative distance-learning system that will allow groups of students to interact with each other remotely and with an intelligent electronic agent that will aid them in their learning has the potential for improving on-line learning. The agent would follow the discussion and interact with the participants when it detects learning trouble of some sort, such as confusion about the problem they are working on or a participant who is dominating the discussion or not interacting with the other participants. In order to recognize problems in the dialogue, we investigated conversational elements that can be utilized as predictors for effective and ineffective interaction between human students. These elements can serve as the basis for student and group models. In this paper, we discuss group interaction during collaborative learning, our representation of participant dialogue, and the statistical models we are using to determine the role being played by a participant at any point in the dialogue and the effectiveness of the group. We also describe student and group models that can be built using conversational elements and discuss one set that we built to illustrate their potential value in collaborative learning.


collaborative learning dialogue modeling distance learning group modeling intelligent agent intelligent tutoring systems student modeling training 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Anderson, J.R. 1990The Adaptive Character of ThoughtErlbaumMahwah, NJGoogle Scholar
  2. Anderson, J.R., Boyle, C.F., Corbett, A., Lewis, M. 1990Cognitive modeling and intelligent tutoringArtificial Intelligence42749CrossRefGoogle Scholar
  3. Ang J., Dhillon R., Krupski A., Shriberg E., Stolcke A. (2002). Prosody-based automatic detection of annoyance and frustration in human–computer dialog. In: Proceedings of ICSLP-2002, Denver, CO, pp. 2037–2040Google Scholar
  4. Batliner A., Noth E., Buckow J., Huber R., Warnke V., Niemann H. (2001). Whence and whither prosody in automatic speech understanding: A case study. In: Proceedings ISCA Tutorial and Research Workshop on Prosody and Speech Recognition and Understanding, Red Bank, NJGoogle Scholar
  5. Belbin, R.M. 2004Management Teams2Butterworth-HeinemannOxfordGoogle Scholar
  6. Benne, K., Sheats, P. 1948Functional roles of group membersJournal of Social Issues44149Google Scholar
  7. Bloom B., S. 1984The two sigma problem: The search for methods of group instruction as effective as one-to-one tutoringEducational Researcher.13416CrossRefGoogle Scholar
  8. Bosma W., André E. (2004). Exploiting emotions to disambiguate dialogue acts. In: Proceedings of the Conference on Intelligent User Interfaces, Portugal, pp. 85–92Google Scholar
  9. Brown, A., Palincsar, A. 1989

    Guided, cooperative learning and individual knowledge acquisition

    Lauren B., Resnick. eds. Knowledge, Learning and Instruction.Lawrence ErlbaumHillsdale, NJ393451
    Google Scholar
  10. Bull, S., Brna, P., Pain, H. 1995Extending the scope of the student modelUser Modeling and User-Adapted Interaction54565CrossRefGoogle Scholar
  11. Burton, M., Brna, P., Pilkington, R., Clarissa.,  2000A Laboratory for the Modelling of CollaborationInternational Journal of Artificial Intelligence in Education1179105Google Scholar
  12. Carletta, J. 1996Assessing agreement on classification tasks: The Kappa statisticComputational Linguistics.22249254Google Scholar
  13. Chan T., Baskin A. (1988). Studying with the prince. The computer as a learning companion. In: Proceedings of the ITS-88 Conference, (Montréal, Canada), pp. 194–200Google Scholar
  14. Chi, M. 1996Constructing self-explanations and scaffolded explanations in tutoringApplied Cognitive Psychology,10S33S49MathSciNetCrossRefGoogle Scholar
  15. Collins, A., Brown, J.S., Newman, S. 1989

    Cognitive apprenticeship: Teaching the craft of reading, writing and mathematics

    Resnick, L.B. eds. Knowing, learning and Instruction: Essays in Honor of Robert Glaser.Lawrence ErlbaumHillsdale, NJ453494
    Google Scholar
  16. Collins, A. 1991

    Cognitive Apprenticeship and Instructional Technology

    Idol, L.Jones, B.F. eds. Educational Values and Cognitive Instruction.Lawrence ErlbaumHillsdale, NJ121138
    Google Scholar
  17. Corbett A.T., Anderson J.R. (1989). ‘Feedback timing and student control in the LISP Intelligent Tutoring System’. In: Proceedings of the Fourth International Conference on AI and Education, pp. 64–72Google Scholar
  18. Czarkowski, M., Kay, J. 2000

    Bringing scrutability to adaptive hypertext teaching

    Gauthier., Frasson., Van, Lehn. eds. Intelligent Tutoring Systems.SpringerBerlin423432
    Google Scholar
  19. Dietterich T.G. (2002). Machine learning for sequential data: A review. In: Caelli T. (ed). Lecture Notes in Computer Science. 2396, 15–30Google Scholar
  20. Felder, R.M., Felder, G.N., Dietz, E.J. 1998A longitudinal study of engineering student performance and retention. V. comparisons with traditionally-taught studentsJournal of Engineering Education.87469480Google Scholar
  21. Flores, F., Graves, M., Hartfield, B., Winograd, T. 1988Computer systems and the design of organizational interactionACM Transactions on Office Information Systems.6153172CrossRefGoogle Scholar
  22. Forbes-Riley K., Litman D.J. (2004). Predicting emotion in spoken dialogue from multiple knowledge sources. In: Proceedings of HLT-NAACL 2004, Boston, MA, pp. 201–208Google Scholar
  23. Goodman B., Iorizzo L. (2000). Learning with reflection: Project PRAXIS. In: Proceedings of the Interservice/Industry Training, Simulation, and Education Conference, Orlando, FLGoogle Scholar
  24. Gaimari R., Soller A.L. (1996). Collaborative learning in an intelligent tutoring system. In: Proceedings of the 1996 Conference on Computer-Supported Cooperative Work Workshop on Approaches for Distributed Learning through Computer-Supported Collaborative Learning, Cambridge, MAGoogle Scholar
  25. Goodman B., Soller A., Linton F., Gaimari R. (1996). [Videotaped study: 3 groups of 4–5 students each solving software system design problems using Object Modeling Technique during a one week course at The MITRE Institute]. Unpublished raw dataGoogle Scholar
  26. Goodman B., Soller A., Linton F., Gaimari R. (1997). Encouraging student reflection and articulation using a learning companion. In: Proceedings of the AI-ED 97 World Conference on Artificial Intelligence in Education, Kobe, Japan, pp. 151–158Google Scholar
  27. Goodman, B., Soller, A., Linton, F., Gaimari, R. 1998Encouraging student reflection and articulation using a learning companionInternational Journal of Artificial Intelligence in Education.9237255Google Scholar
  28. Goodman, B., Geier, M., Haverty, L., Linton, F., McCready, R. 2001

    A framework for asynchronous collaborative learning and problem-solving

    Moore, J.Redfield, C.L.Johnson, W.L. eds. Artificial Intelligence in Education.IOS PressAmsterdam188199
    Google Scholar
  29. Goodman B., Hitzeman J., Linton F., Ross H. (2003a). Towards intelligent agents for collaborative learning: recognizing the role of dialogue participants. In: Proceedings of the International Conference on User Modeling, Johnstown, PA, pp. 363–367Google Scholar
  30. Goodman, B., Gaimari, R., Zarrella, J., Linton, F.,  et al. 2003

    An empirical analysis of learner discourse

    Hoppe, H.U. eds. Proceedings of the 10th International Conference on Artificial Intelligence in Education: Artificial Intelligence in Education.IOS PressAmsterdam416418
    Google Scholar
  31. Hmelo-Silver C.E. (2002). Collaborative ways of knowing: Issues in facilitation. In: Stahl G. (ed). Proceedings of CSCL 2002, Boulder, CO, pp. 199–208Google Scholar
  32. Hillard D., Ostendorf M., Shriberg E. (2003). Detection of agreement vs. disagreement in meetings: Training with unlabeled data. In: Proceedings of HLT-NAACL Conference, Edmonton, Canada, pp. 34–36Google Scholar
  33. Hirschberg J., Litman D.J., Swerts M. (2000). Generalizing prosodic prediction of speech recognition errors. In: Proceedings of the 6th International Conference of Spoken Language Processing (ICSLP-2000), Beijing, China, pp. 615–618Google Scholar
  34. Jarboe, S. 1996

    Procedures for enhancing group decision making

    Hirokawa, B.Poole, M. eds. Communication and Group Decision Making.Sage PublicationsThousand Oaks, CA345383
    Google Scholar
  35. Jonassen, D., Land, S. 2000Theoretical Foundations of Learning EnvironmentsErlbaumMahwah, NJGoogle Scholar
  36. Johnson, D., Johnson, R., Holubec, E.J. 1990Circles of Learning: Cooperation in the ClassroomInteraction Book CompanyEdina, MNGoogle Scholar
  37. Jokinen K., Hurtig T., Hynnä K., Kanto K., Kaipainen M., Kerminen A. (2001). Self-organizing dialogue management. In: Proceedings of the 2nd Workshop on Neural Networks and Natural Language Processing, Natural Language Pacific Rim Symposium (NLPRS), Tokyo, Japan, pp. 78–85Google Scholar
  38. Katz, S., Aronis, J., Creitz, C. 1999

    Modeling pedagogical interactions with machine learning

    Lajoie, S.P.Vivet, M. eds. Artificial Intelligence in Education.IOS PressAmsterdam543550
    Google Scholar
  39. Katz, S., O’Donnell, G., Kay, H. 2000approach to analyzing the role and structure of reflective dialogueInternational Journal of Artificial Intelligence in Education11320343Google Scholar
  40. Kay J. (1998). Scrutable User Modeling Shell for User-adapted Interaction. Ph.D. Thesis Basser Department of Computer Science. University of Sydney, AustraliaGoogle Scholar
  41. Kneser, C., Pilkington, R., Treasure-Jones, T. 2001The tutor’s role: An investigation of the power of exchange structure analysis to identify different roles in CMC seminarsInternational Journal of Artificial Intelligence in Education126384Google Scholar
  42. Lesgold, A., Katz, S., Greenberg, L., Hughes, E., Eggan, G. 1992

    Extensions of intelligent tutoring paradigms to support collaborative learning

    Dijkstra, S.Krammer, H.Merrienboer, J. eds. Instructional Models in Computer-Based Learning Environments.SpringerBerlin291311
    Google Scholar
  43. Linton F., Goodman B., Gaimari R., Zarrella J., Ross H. (2003). Student modeling for an intelligent agent in a collaborative learning environment. In: Proceedings of the International Conference on User Modeling, Johnstown, PA, pp. 342–352Google Scholar
  44. Litman D.J., Hirschberg J., Swerts M. (2000). Predicting automatic speech recognition performance using prosodic cues. In: Proceedings of the First Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL’00), Seattle, WA, pp. 218–225Google Scholar
  45. Lund K., Baker M., Baron M. (1996). Modeling dialogue and beliefs as a basis for generating guidance in a CSCL environment. In: Proceedings of the ITS-96 Conference, Montreal, pp. 206–614Google Scholar
  46. McManus, M., Aiken, R. 1995Monitoring computer-based problem-solvingJournal of Artificial Intelligence in Education.6307336Google Scholar
  47. Morales, R., Pain, H., Conlon, T. 2000

    Understandable learner models for a sensorimotor control task

    Gauthier., Frasson., Van, Lehn. eds. Intelligent Tutoring Systems.SpringerBerlin222231
    Google Scholar
  48. Mühlenbrock, M., Tewissen, F., Hoppe, H.U. 1998A framework system for intelligent support in open distributed learning environmentsInternational Journal of Artificial Intelligence in Education.9256274Google Scholar
  49. Rabiner L.R. (1989). A tutorial on hidden Markov models. In: Proceedings of the IEEE, Vol. 77, pp. 257–286Google Scholar
  50. Rehg W., McBurney P., Parsons S. (2001). Computer decision support systems for public argumentation: criteria for assessment. In: Hansen H.V., Tindale C.W., Blair J.A., Johnson R.H. (ed). Argumentation and its Applications. Proceedings of the Fourth Biennial Conference of the Ontario Society for the Study of Argumentation, OSSA 2001Google Scholar
  51. Rumbaugh, J., Blaha, M., Premerlani, W., Eddy, F., Lorensen, W. 1991Object-Oriented modeling and designPrentice HallEnglewood Cliffs, NJGoogle Scholar
  52. Samuel K., Carberry S., Vijay-Shanker K. (1998). Dialogue act tagging with transformation-based learning. In: Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1150–1156Google Scholar
  53. Searle, J. 1969DIALOGUE ACTS: An Essay in the Philosophy of LanguageCambridge University PressLondonGoogle Scholar
  54. Searle, J. 1996

    A taxonomy of illocutionary acts

    Martinich, A. eds. The Philosophy of Language3Oxford University PressNew York
    Google Scholar
  55. Shriberg, E., Bates, R., Stolcke, A., Taylor, P., Jurafsky, D., Ries, K., Coccaro, N., Martin, R., Meteer, M., Van Ess-Dykema, C. 1998Can prosody aid the automatic classification of dialog acts in conversational speechLanguage and Speech.41439487Google Scholar
  56. Shriberg E., Stolcke A., Baron D. (2001). Can prosidy aid the automatic processing of multi-party meetings? Evidence from predicting punctuation, disfluencies, and overlapping speech. In: Proceedings of the ISCA Tutorial and Research Workshop on Prosody in Speech Recognition and Understanding, Red Bank, NJ, pp. 139–146Google Scholar
  57. Shute, V.J. 1990Rose Garden Promises of Intelligent Tutoring Systems: Blossom or Thorn? Presented at the Space Operations and Research (SOAR) SymposiumAlbuquerqueNMGoogle Scholar
  58. Singley M.K., Singh M., Fairweather P., Farrell R., Swerling S. (2000). Algebra jam: Supporting teamwork and managing roles in a collaborative learning environment. In: Proceedings of Computer Supported Collaborative Work 2000, Philadelphia, PA, pp. 145–154Google Scholar
  59. Soller A. (2002). Computational Analysis of Knowledge Sharing in Collaborative Distance Learning. Doctoral Dissertation. Department of Computer Science, University of PittsburghGoogle Scholar
  60. Soller A. (2003). Personal communicationGoogle Scholar
  61. Soller, A. 2004Computational modeling and analysis of knowledge sharing in collaborative distance learningUser Modeling and User-Adapted Interaction.14351381CrossRefGoogle Scholar
  62. Soller A.L. (1997a). An intelligent CSCL communication interface. In: Proceedings of AI-ED 97 Workshop IV: Collaborative Learning/Working Support System with Networking, Kobe, Japan, pp. 94–95Google Scholar
  63. Soller A.L. (1997b). Back to the drawing board: Explaining causal relationships in an argumentation-based ITS. In: Proceedings of the AI-ED 97 World Conference on Artificial Intelligence in Education, Kobe, Japan, pp. 231–238Google Scholar
  64. Soller A., Goodman B., Linton F., Gaimari R. (1998). Promoting effective peer interaction in an intelligent collaborative learning environment. In: Proceedings of the Fourth International Conference on Intelligent Tutoring Systems (ITS 98), San Antonio, TX, pp. 186–195Google Scholar
  65. Soller, A., Lesgold, A. 2000Modeling the process of collaborative learning. International workshop on new technologies in Collaborative LearningAwaji-YumebutaiJapanGoogle Scholar
  66. Soller A., Lesgold A., Linton F., Goodman B. (1999a). What makes peer interaction effective? Modeling effective communication in an intelligent CSCL. In: Proceedings of the 1999 AAAI Fall Symposium: Psychological Models of Communication in Collaborative Systems, Cape Cod, MA, pp. 116–123Google Scholar
  67. Soller A., Lesgold A. (1999). Analyzing peer dialogue from an active learning perspective. Proceedings of the AI-ED 99 Workshop: Analysing Educational Dialogue Interaction: Towards Models that Support Learning, LeMans, France, pp. 63–71Google Scholar
  68. Soller A., Linton F., Goodman B., Lesgold A. (1999b). Toward intelligent analysis and support of collaborative learning interaction. In: Proceedings of the Ninth International Conference on Artificial Intelligence in Education, LeMans, France, pp. 75–82Google Scholar
  69. Soller, A., Wiebe, J., Lesgold, A. 2002

    A machine learning approach to assessing knowledge sharing during collaborative learning activities

    Stahl, G. eds. Proceedings of CSCL 2002.BoulderCO128137
    Google Scholar
  70. Soller, A.L. 2001Supporting social interaction in an intelligent collaborative learning systemInternational Journal of Artificial Intelligence in Education.124062Google Scholar
  71. Stevens R., Ikeba J., Casillas A., Palacio-Cayetano J., Clyman S. (1999). Artificial neural network-based performance assessments, Computers in Human Behavior, 15, pp. 295–313Google Scholar
  72. Tesluk, P., Mathieu, J.E., Zaccaro, S.J., Marks, M. 1997

    Task and aggregation issues in the analysis and assessment of team performance

    Brannick, M.T.Salas, E.Prince, C. eds. Team Performance Assessment and Measurement.Lawrence ErlbaumMahwah, NJ197224
    Google Scholar
  73. de Vicente A., Bouwer A., Pain H. (1999). Initial impressions on using the DISCOUNT scheme. In: Proceedings of the Workshop on Analysing Educational Dialogue Interaction (AIED 99 Workshop), LeMans, France, pp. 87–94Google Scholar
  74. Waibel, A., Bett, M., Finke, M., Stiefelhagen, R. 1998

    Meeting browser: Tracking and summarizing meetings

    Penrose, D.E.M. eds. Proceedings of the Broadcast News Transcription and Understanding Workshop.Morgan KaufmannLansdowne, VA281286
    Google Scholar
  75. Walker M.A., Wright J., Langkilde I. (2000). Using natural language processing and discourse features to identify understanding errors in a spoken dialogue system. In: Proceedings of the International Conference on Machine Learning, Stanford, CA, pp. 1111–1118Google Scholar
  76. Winter M., McCalla G. (2003). An analysis of group performance in terms of the functional knowledge and teamwork skills of group members. In: Proceedings of the UM2003 Workshop on User and Group Models for Web-based Adaptive Collaborative Environments, Johnstown, PA, pp. 35–44Google Scholar
  77. Wu, A., Farrell, R., Singley, M. 2002

    Scaffolding Group Learning in a Collaborative Networked Environment

    Stahl, G. eds. Proceedings of CSCL 2002.BoulderCO245254
    Google Scholar

Copyright information

© Springer 2005

Authors and Affiliations

  • Bradley A. Goodman
    • 1
  • Frank N. Linton
    • 1
  • Robert D. Gaimari
    • 1
  • Janet M. Hitzeman
    • 1
  • Helen J. Ross
    • 1
  • Guido Zarrella
    • 1
  1. 1.The MITRE CorporationBedfordUSA

Personalised recommendations