Abstract
Assessment of difficulties within group processes, especially through automatic means, is a problem of great interest to the broader CSCL community. Group difficulties can be revealed through interaction processes that occur during group work. Whether these patterns are encoded in speech recorded from face-to-face interactions or in text from on-line interactions, the language communication that flows between group members is an important key to understanding how better to support group functions and therefore be in a better position to design effective group learning environments. With the capability of monitoring and then influencing group processes when problems are detected, it is possible to intervene in order to facilitate the accomplishment of a higher quality product. In this chapter we address this research problem of monitoring group work processes in a context where project course instructors are making assessments of student group work. Thus, our purpose is to support those instructors in their task. We describe the mixed methods approach that we took, which combines both an interview study and a classroom study. Three research questions are answered: (1) What do instructors want to know about their student groups? (2) Is the desired information observable, and can it be reliably tracked by human annotators? (3) Can the desired information be automatically tracked using machine learning techniques to produce a summary report that instructors can use? Based on interviews with nine instructors, we identified five process assessment categories with subcategories at the group and individual level: namely, goal setting, group and individual progress, knowledge contribution, participation, and teamwork. We verified that these assessment categories can be reliably coded during group meetings with a reliability of r = 0.80 at the group level and r = 0.64 at the individual level using carefully constructed human assessment instruments. We present work in progress towards automation of this assessment framework.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Adams, R. S., Turns, J., & Atman, C. J. (2003). Educating effective engineering designers: The role of reflective practice. Design Studies, 24(3), 275–294.
Beskow, J., & Sjlander, K. (2000). Wavesurfer–an open source speech tool. In Proceedings of ICSLP, Beijing, 464–467.
Chen, M. (2003). Visualizing the pulse of a classroom. In Proceedings of international multimedia conference (pp. 555–561). Berkeley: ACM Press.
Dong, A., Hill, W. A., & Agogino, A. M. (2004). A document analysis technique for characterizing design team performance. Journal of Mechanical Design, 126(3), 378–385.
Donmez, P., Rose, C. P., Stegmann, K., Weinberger, A., & Fischer, F. (2005). Supporting CSCL with automatic corpus analysis technology. In T. Koschmann, D. Suthers, & T.-W. Chan (Eds.), Proceedings of computer supported collaborative learning (pp. 125–134). Taipei: Laurence Earlbaum Associates.
Drach-Zahavy, A., & Somech, A. (2001). Understanding team innovation: The role of team processes and structures. Group Dynamics, 5(2), 111–123.
Dutson, A. J., Todd, R. H., Magleby, S. P., & Sorensen, C. D. (1997). A review of literature on teaching design through project-oriented capstone courses. Journal of Engineering Education, 76(1), 17–28.
Faidley, J., et al. (2000). How are we doing? Methods of assessing group processing in a problem-based learning context. In D. H. Evensen & C. E. Hmelo (Eds.), Problem-based learning: A research perspective on learning interactions (pp. 109–135). NJ: Erlbaum.
Fussell, S. R., et al. (1998). Coordination, overload and team performance: Effects of team communication strategies. In Proceeding of conference on computer-supported cooperative work, Seattle (pp. 275–284). New York: ACM Press. 14–18 November 1998.
Gladstein, D. L. (1984). Groups in context: A model of task group effectiveness. Administrative Science Quarterly, 29, 499–517.
Hackman, R. (1987). The design of work teams. In J. Lorsch (Ed.), Handbook of organizational behavior. Englewood Cliffs: Prentice-Hall.
Hmelo-Silver, C. E. (2004). Problem based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266.
Joshi, M., & Rosé, C. P. (2007). Using transactivity in conversation summarization in educational dialog. In Proceedings of the SLaTE workshop on speech and language technology in education.
Karau, S., & Williams, K. (1993). Social loafing: A meta-analytic review and theoretical integration. Journal of Personality and Social Psychology, 65, 681–706.
Kay, J., Maisonneuve, N., Yacef, K., & Reimann, P. (2006). Wattle tree: What’ll it tell us?, University of Sydney, Technical Report, p 582.
Kreijns, et al. (2002). The sociability of computer-supported collaborative learning environments. Journal Of Education Technology & Society, 5(1), 8–22.
Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. New York: Cambridge University Press. ISBN 0521423740.
Madan, A., Caneel, R., & Pentland, A. (2004). Group media: distributed multimodal interfaces. In Proceedings of sixth international conference on multimodal interfaces ICMI’04.
McGrath, J. (1984). Groups: Interaction and performance. Englewood Cliffs: Prentice-Hall.
McLaren, B., Scheuer, O., De Laat, M., Hever, R., de Groot, R., & Rosé, C. P. (2007). Using machine learning techniques to analyze and support mediation of student e-discussions. In Proceedings of artificial intelligence in education.
Meier, A., Spada, H., & Rummel, N. (2007). Evaluating collaboration: A rating scheme for assessing the quality of collaborative process. International Journal of Computer-Supported Collaborative Learning, 2, 63–86.
Meloth, M. S., & Deering, P. D. (1999). The role of the teacher in promoting cognitive processing during collaborative learning. In O’Donnell & King (Eds.), Cognitive perspectives on peer learning. Hillsdale: Lawrence Erlbaum Associates.
Pea, R. D. (1993). Practices of distributed intelligence and designs for education. In G. Salomon (Ed.), Distributed cognitions (pp. 47–87). New York: Cambridge University Press.
Pennebaker, J. W., et al. (2008). Linguistic inquiry and word count. http://www.liwc.net/. Retrieved 24 April 2008.
Rosé, C., Gweon, G., Arguello, J., Finger, S., Smailagic, A., & Siewiorek, D. (2007). Towards an Interactive Assessment Framework for Engineering Design Project Based Learning. In Proc. DETC 2007.
Rosé, C. P., Wang, Y. C., Cui, Y., Arguello, J., Stegmann, K., Weinberger, A., Fischer, F. (2008) Analyzing collaborative learning processes automatically: exploiting the advances of computational linguistics in computer-supported collaborative learning. International Journal of Computer Supported Collaborative Learning.
Rousseau, V., Aube, C., & Savoie, A. (2006). Teamwork behaviors: A review and an integration of frameworks. Small Group Research, 37(5), 540–570.
Rienks, R. J., Zhang, D., Gatica-Perez, D., & Post, W. (2006). Detection and application of influence rankings in small group meetings. In Proceedings of eighth international conference on multimodal interfaces ICMI’06.
Salomon, G. (1993). No distribution without individual cognition: A dynamic interactional view. In G. Salomon (Ed.), Distributed cognitions (pp. 111–139). Cambridge: Cambridge University Press.
Shneiderman, B. (1992). Tree visualization with tree-maps: 2-d space-filling approach. ACM Transactions on Graphics, 11(1), 92–99.
Steiner, I. D. (1972). Group process and productivity. New York: Academic.
Strijbos, J.W. (2004). The effect of roles on computer-supported collaborative learning. Ph. D dissertation, Open University of the Netherlands, The Netherlands
Smith, M., & Fiore, A. (2001). Visualization components for persistent conversations. In Proceeding of conference on human factors on computing systems, Seattle (pp. 136–143). New York: ACM Press. March 31 – April 5, 2001.
Weinberger, A., & Fischer, F. (2005). A framework to analyze argumentative knowledge construction in computer-supported collaborative learning. Computers & Education, 46(1), 71–95.
Witten, I. H., & Frank, E. (2005). Data mining: Practical machine learning tools and techniques (2nd ed.). Elsevier: San Francisco.
Acknowledgments
We are deeply grateful to the instructors that participated in our interview study as well as the students and instructors who participated in the classroom study. Without their patience and participation, this work would not have been possible. This research was supported in part by NSF EEC grant number 064848.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer Science+Business Media, LLC
About this chapter
Cite this chapter
Gweon, G., Jun, S., Lee, J., Finger, S., Rosé, C.P. (2011). A Framework for Assessment of Student Project Groups On-Line and Off-Line. In: Puntambekar, S., Erkens, G., Hmelo-Silver, C. (eds) Analyzing Interactions in CSCL. Computer-Supported Collaborative Learning Series, vol 12. Springer, Boston, MA. https://doi.org/10.1007/978-1-4419-7710-6_14
Download citation
DOI: https://doi.org/10.1007/978-1-4419-7710-6_14
Published:
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4419-7709-0
Online ISBN: 978-1-4419-7710-6
eBook Packages: Humanities, Social Sciences and LawEducation (R0)