A rating scheme for assessing the quality of computer-supported collaboration processes

Article

Abstract

The analysis of the process of collaboration is a central topic in current CSCL research. However, defining process characteristics relevant for collaboration quality and developing instruments capable of assessing these characteristics are no trivial tasks. In the assessment method presented in this paper, nine qualitatively defined dimensions of collaboration are rated quantitatively: sustaining mutual understanding, dialogue management, information pooling, reaching consensus, task division, time management, technical coordination, reciprocal interaction, and individual task orientation. The data basis for the development of these dimensions was taken from a study in which students of psychology and medicine collaborated on a complex patient case via a desktop-videoconferencing system. A qualitative content analysis was performed on a sample of transcribed collaboration dialogue. The insights from this analysis were then integrated with theoretical considerations about the roles of communication, joint information processing, coordination, interpersonal relationship, and motivation in the collaboration process. The resulting rating scheme was applied to process data from a new sample of 40 collaborating dyads. Based on positive findings on inter-rater reliability, consistency, and validity from this evaluation, we argue that the new method can be recommended for use in different areas of CSCL.

Keywords

Assessment Collaboration Communication Coordination Group information processing Interpersonal relationship Motivation Rating scheme Videoconferencing 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Anderson, A. H., O’Malley, C., Doherty-Sneddon, G., Langton, S., Newlands, A., Mullin, J., et al. (1997). The impact of VMC on collaborative problem solving: An analysis of task performance, communicative process, and user satisfaction. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 133–156). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  2. Angiolillo, J. S., Blanchard, H. E., Israelski, E. W., & Mané, A. (1997). Technology constraints of video-mediated communication. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 51–74). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  3. Barron, B. (2000). Achieving coordination in collaborative problem-solving groups. Journal of the Learning Sciences, 9, 403–436.CrossRefGoogle Scholar
  4. Carell, A., Herrman, T., Kienle, A., & Menold, N. (2005). Improving the coordination of collaborative learning with process models. In T. Koschmann, D. Suthers, & T. W. Chan (Eds.), Proceedings of the CSCL 2005 (pp. 18–27). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  5. Clark, H. H. (1996). Using language. Cambridge MA: Cambridge University Press.Google Scholar
  6. Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In L. B. Resnick, J. M. Levine, & S. D. Teasley (Eds.), Perspectives on socially shared cognition (pp. 127–148). Washington, DC: American Psychological Association.CrossRefGoogle Scholar
  7. Clark, D., & Sampson, V.D. (2005). Analyzing the quality of argumentation supported by personally-seeded discussions. In T. Koschmann, D. Suthers, & T. W. Chan (Eds.), Proceedings of the CSCL 2005 (pp. 76–85). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  8. De Wever, B., Schellens, T., Valcke, M., & Van Keer, H. (2006). Content analysis schemes to analyze transcripts of online asynchronous discussion groups: A review. Computers and Education, 46, 6–28.CrossRefGoogle Scholar
  9. Deutsch, M. (2003). Cooperation and conflict. A personal perspective on the history of the social psychological study of conflict resolution. In M. A. West, D. Tjosvold, & K. G. Smith (Eds.), International handbook of organizational teamwork and cooperative working (pp. 9–43). Chichester, UK: Wiley.Google Scholar
  10. Dillenbourg, P. (1999). Introduction: What do you mean by “collaborative learning”? In P. Dillenbourg (Ed.), Collaborative learning. Cognitive and computational approaches (pp. 1–19). Amsterdam: Pergamon.Google Scholar
  11. Dillenbourg, P., Baker, M., Blaye, A., & O’Malley, C. (1995). The evolution of research on collaborative learning. In P. Reimann & H. Spada (Eds.), Learning in humans and machines: Towards an interdisciplinary learning science (pp. 189–211). Oxford: Pergamon.Google Scholar
  12. Dönmez, P., Rose, C., Stegmann, K., Weinberger, A., & Fischer, F. (2005). Supporting CSCL with automated corpus analysis technology. In T. Koschmann, D. Suthers, & T. W. Chan (Eds.), Proceedings of the CSCL 2005 (pp. 125–134). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  13. Erkens, G., Jaspers, J., Prangsma, M., & Kanselaar, G. (2005). Coordination processes in computer supported collaborative writing. Computers in Human Behavior, 21, 463–486.CrossRefGoogle Scholar
  14. Fischer, F., & Mandl, H. (2003). Being there or being where? Videoconferencing and cooperative learning. In H. van Oostendorp (Ed.), Cognition in a digital world (pp. 205–223). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  15. Heckhausen, H. (1989). Motivation und Handeln. [Motivation and behavior]. Berlin Heidelberg New York: Springer.Google Scholar
  16. Hermann, F., Rummel, N., & Spada, H. (2001). Solving the case together: The challenge of net-based interdisciplinary collaboration. In P. Dillenbourg, A. Eurelings, & K. Hakkarainen (Eds.), Proceedings of the first European conference on computer-supported collaborative learning (pp. 293–300). Maastricht: McLuhan Institute.Google Scholar
  17. Hinsz, V. B., Tindale, R. S., & Vollrath, D. A. (1997). The emerging conceptualization of groups as information processors. Psychological Bulletin, 121(1), 43–64.CrossRefGoogle Scholar
  18. Janis, I. L. (1982). Groupthink. Boston: Houghton Mifflin.Google Scholar
  19. Järvelä, S., & Häkkinen, P. (2003). The levels of web-based discussions: Using perspective-taking theory as an analytical tool. In H. van Oostendorp (Ed.), Cognition in a digital world (pp. 77–95). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  20. Johnson, D. W., & Johnson, R. T. (2003). Training for cooperative group work. In M. A. West, D. Tjosvold, & K. G. Smith (Eds.), International handbook of organizational teamwork and cooperative working (pp. 167–183). Chichester, UK: Wiley.Google Scholar
  21. Jucks, R., Bromme, R., & Runde, A. (2003). Audience Design von Experten in der netzgestützten Kommunikation: Die Rolle von Heuristiken über das geteilte Vorwissen. (Audience design of experts in net-based communication: The role of heuristics about shared knowledge). Zeitschrift für Psychologie, 211(2), 60–74.CrossRefGoogle Scholar
  22. Kapur, M., Voiklis, J., & Kinzer, C. K. (2005). Problem solving as a complex, evolutionary activity: A methodological framework for analyzing problem solving processes in a computer-supported collaborative environment. In T. Koschmann, D. Suthers, & T. W. Chan (Eds.), Proceedings of the CSCL 2005 (pp. 252–261). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  23. Kerlinger, F. N., & Lee, H. B. (2000). Foundations of behavioral research. Fort Worth: Harcourt College.Google Scholar
  24. Kneser, C., & Ploetzner, R. (2001). Collaboration on the basis of complementary domain knowledge: Observed dialogue structures and their relation to learning success. Learning and Instruction, 11(1), 53–83.CrossRefGoogle Scholar
  25. Koschmann, T., Zemel, A., Conlee-Stevens, M., Young, N., Robbs, J., & Barnhart, A. (2003). Problematizing the problem. In B. Wasson, S. Ludvigsen, & U. Hoppe (Eds.), Designing for change (pp. 37–46). Dordrecht: Kluwer.Google Scholar
  26. Larson, J. R., & Christensen, C. (1993). Groups as problem-solving units: Toward a new meaning of social cognition. British Journal of Social Psychology, 32, 5–30.Google Scholar
  27. Lee, E. Y. C., Chan, C. K. K., & van Aalst, J. (2006). Students assessing their own collaborative knowledge building. International Journal of Computer-supported Collaborative Learning, 1, 277–307.CrossRefGoogle Scholar
  28. Malone, T. W., & Crowston, K. (1990). What is coordination theory and how can it help design cooperative work systems? Proceedings of the Conference on Computer-Supported Cooperative Work (pp. 357–370). Los Angeles, CA.Google Scholar
  29. Malone, T. W., & Crowston, K. (1994). The interdisciplinary study of coordination. ACM Computing Surveys, 26(1), 87–119.CrossRefGoogle Scholar
  30. Mayring, P. (2003). Qualitative Inhaltsanalyse. Grundlagen und Techniken (Qualitative content analysis. Foundations and techniques). Weinheim: Beltz.Google Scholar
  31. Moreland, R. L., & Myaskovsky, L. (2000). Exploring the performance benefits of group training: Transactive memory or improved communication? Organizational Behavior and Human Decision Processes, 82(1), 117–133.CrossRefGoogle Scholar
  32. Nickerson, R. S. (1999). How we know—and sometimes misjudge—what others know: Imputing one’s own knowledge to others. Psychological Bulletin, 125(6), 737–759.CrossRefGoogle Scholar
  33. Nurmela, K., Palonen, T., Lehtinen, E., & Hakkarainen, K. (2003). Developing tools for analyzing CSCL process. In B. Wasson, S. Ludvigsen, & U. Hoppe (Eds.), Designing for change (pp. 333–342). Dordrecht: Kluwer.Google Scholar
  34. O’Conaill, B., & Whittaker, S. (1997). Characterizing, predicting, and measuring video-mediated communication: A conversational approach. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 107–132). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  35. Postmes, T., Spears, R., & Cihangir, S. (2001). Quality of decision making and group norms. Journal of Personality and Social Psychology, 80(6), 918–930.CrossRefGoogle Scholar
  36. Prins, F. J., Sluijsmans, D. M. A., Kirschner, P. A., & Strijbos, J. -W. (2005). Formative peer assessment in a CSCL environment: A case study. Assessment and Evaluation in Higher Education, 30(4), 417–444.Google Scholar
  37. Rummel, N., & Spada, H. (2005a). Instructional support for collaboration in desktop videoconferencing settings. How it can be achieved and assessed. In R. Bromme, F. W. Hesse, & H. Spada (Eds.), Barriers and biases in computer-mediated knowledge communication— and how they may be overcome (pp. 59–88). Berlin Heidelberg New York: Springer.CrossRefGoogle Scholar
  38. Rummel, N., & Spada, H. (2005b). Learning to collaborate: An instructional approach to promoting problem-solving in computer-mediated settings. Journal of the Learning Sciences, 14(2), 201–241.CrossRefGoogle Scholar
  39. Rummel, N., Spada, H., & Hauser, S. (2006). Learning to collaborate in a computer-mediated setting: Observing a model beats learning from being scripted. In: S. A. Barab, K. E. Hay, & D. T. Hickey (Eds.), Proceedings of the International Conference of the Learning Sciences 2006 (pp. 634–640). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  40. Sacks, H., Schegloff, E., & Jefferson, G. (1974). A simplest systematic for the organization of turn-taking in conversation. Language, 50, 696–753.CrossRefGoogle Scholar
  41. Sosa y Fink, S. (2003). Merkmale gelungener Kooperation. Eine qualitative Analyse netzgestützter Zusammenarbeit. (Characteristics of successful cooperation. A qualitative analysis of net-based collaboration.) Diploma thesis, Albert-Ludwigs-Universität, Freiburg.Google Scholar
  42. Spada, H., Meier, A., Rummel, N., & Hauser, S. (2005). A new method to assess the quality of collaborative process in CSCL. In T. Koschmann, D. Suthers, & Chan, T.W. (Eds.), Proceedings of the CSCL 2005 (pp. 622–631). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  43. Stasser, G., Stewart, D., & Wittenbaum, G. (1995). Expert roles and information exchange during discussion: The importance of knowing who knows what. Journal of Experimental Social Psychology, 31, 244–265.CrossRefGoogle Scholar
  44. Stasser, G., & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during group discussion. Journal of Personality and Social Psychology, 48, 1467–1478.CrossRefGoogle Scholar
  45. Steiner, I. D. (1972). Group process and productivity. New York: Academic.Google Scholar
  46. Tindale, R. S., Kameda, T., & Hinsz, V. B. (2003). Group decision making. In M. A. Hogg & J. Cooper (Eds.), Sage handbook of social psychology (pp. 381–403). London: Sage.Google Scholar
  47. Webb, N. M. (1989). Peer interaction and learning in small groups. International Journal of Education Research, 13, 21–39.CrossRefGoogle Scholar
  48. Wegner, D. M. (1987). Transactive memory: A contemporary analysis of the group mind. In B. Mullen & G. R. Goethals (Eds.), Theories of group behavior (pp. 185–208). Berlin Heidelberg New York: Springer.Google Scholar
  49. Weinberger, A., & Fischer, F. (2006). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Computers and Education, 46, 71–95.CrossRefGoogle Scholar
  50. Whittaker, S., & O’Conaill, B. (1997). The role of vision in face-to-face and mediated communication. In K. E. Finn, A. J. Sellen, & S. B. Wilbur (Eds.), Video-mediated communication (pp. 23–50). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  51. Wirtz, M., & Caspar, F. (2002). Beurteilerübereinstimmung und Beurteilerreliabilität. (Inter-rater agreement and inter-rater reliability). Göttingen: Verlag für Psychologie.Google Scholar
  52. Wittenbaum, G. M., Vaughan, S. I., & Stasser, G. (1998). Coordination in task performing groups. In R. S. Tindale et al. (Eds.), Theory and research on small groups (pp. 177–204). New York: Plenum.Google Scholar
  53. Zumbach, J., Schönemann, J., & Reimann, P. (2005). Analyzing and supporting cooperative computer-mediated communication. In T. Koschmann, D. Suthers, & Chan, T.W. (Eds.), Proceedings of the CSCL 2005 (pp. 758–767). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar

Copyright information

© International Society of the Learning Sciences, Inc.; Springer Science+ Business Media, LLC 2007

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of FreiburgFreiburgGermany

Personalised recommendations