Abstract
Per definition, CSCL research deals with the data of individuals nested in groups, and the influence of a specific learning setting on the collaborative process of learning. Most well-established statistical methods are not able to analyze such nested data adequately. This article describes the problems which arise when standard methods are applied and introduces multilevel modelling (MLM) as an alternative and adequate statistical approach in CSCL research. MLM enables testing interactional effects of predictor variables varying within groups (for example, the activity of group members in a chat) and predictors varying between groups (for example, the group homogeneity created by group members’ prior knowledge). So it allows taking into account that an instruction, tool or learning environment has different but systematic effects on the members within the groups on the one hand and on the groups on the other hand. The underlying statistical model of MLM is described using an example from CSCL. Attention is drawn to the fact that MLM requires large sample sizes which are not provided in most CSCL research. A proposal is made for the use of some analyses which are useful.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bonito, J. A. (2002). The analysis of participation in small groups: Methodological and conceptual issues related to interdependence. Small Group Research, 33, 412–438.
Bonito, J. A., & Lambert, B. L. (2005). Information similarity as a moderator of the effect of gender on participation in small groups: A multilevel analysis. Small Group Research, 36, 139–165.
Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models. Newbury Park, CA: Sage.
Burstein, L. (1978). Assessing differences between grouped and individual-level regression coefficients. Sociological Methods & Research, 7, 5–28.
Burstein, L. (1980). The analysis of multilevel data in educational research and evaluation. Review of Research in Education, 8, 158–232.
Burstein, L., Kim, S. S., & Delandshere, G. (1989). Multilevel investigations of systematically varying slopes: Issues, alternatives, and consequences. In D. Bock (Ed.) Multilevel analysis of educational data (pp. 233–279). San Diego: Academic.
Chiu, M. M., & Khoo, L. (2003). Rudeness and status effects during group problem solving: Do they bias evaluations and reduce the likelihood of correct solutions. Journal of Educational Psychology, 95, 506–523.
Chiu, M. M., & Khoo, L. (2005). A new method for analyzing sequential processes: Dynamic multilevel modeling. Small Group Research, 36, 600–631.
Cress, U. (2005). Ambivalent effect of member portraits in virtual groups. Journal of Computer-Assisted Learning, 21, 281–291.
De Wever, B., Van Keer, H., Schellens, T., & Valcke, M. (2007). Applying multilevel modelling to content analysis data: Methodological issues in the study of role assignment in asynchronous discussion groups. Leaning & Instruction, 17, 436–447.
Hox, J. J. (2002). Multilevel analysis: techniques and applications. Mahwah, NJ: Lawrence Erlbaum Associates.
Hron, A., Hesse, F. W., Cress, U., & Giovis, C. (2000). Implicit and explicit dialogue structuring in virtual learning groups. British Journal of Educational Psychology, 70, 53–64.
Kashy, D. A., & Kenny, D. A. (2000). The analysis of data from dyads and groups. In H. T. Reis, & C. M. Judd (Eds.) Handbook of research methods in social and personality psychology. Cambridge, UK: Cambridge University Press.
Kenney, D. A., & Judd, C. M. (1986). Consequences of violating the independence assumption in analysis of variance. Psychological Bulletin, 99, 422–431.
Kenny, D. A., Kashy, D. A., & Bolger, N. (1998). Data analysis in social psychology. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.) The handbook of social psychology (4th ed., Vol. 1, pp. 233–265). New York: McGraw Hill.
Kenny, D. A., Mannetti, L., Pierro, A., Livi, S., & Kashy, D. A. (2002). The statistical analysis of data from small groups. Journal of Personality and Social Psychology, 83, 126–137.
Kimmerle, J. & Cress (2008). Group Awareness and Self-Presentation in Computer-Supported Information Exchange. International Journal of Computer-Supported Collaborative Learning DOI 10.1007/s11412-007-9027-z.
Kreft, I. (1996). Are multilevel techniques necessary? An overview, including simulation studies. Retrieved March, 20, 2007 from http://www.calstatela.edu/faculty/ikreft/quarterly/quarterly.html.
Maas, C. J. M., & Hox, J. (2005). Sufficient samples sizes for multilevel modeling. Methodology, 1, 86–92.
Marsh, H. W., & Hau, K. T. (2003). Big-fish-little-pond effect on academic self-concept: A cross-cultural (26-Country) test of the negative effects of academically selective schools. American Psychologist, 58(5), 364–376.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.
Paccagnella, O. (2006). Centering or not centering in multilevel models? The role of the group mean and the assessment of group effects. Evaluation Review, 30, 66–85.
Piontkowski, U., Keil, W., & Hartmann, J. (2006). Analyseebenen und Dateninterdependenz in der Kleingruppenforschung am Beispiel netzbasierter Wissensintegration. Zeitschrift für Sozialpsychologie, 37, 41–50.
Raudenbush, S., & Bryk, A. (2002). Hierarchical linear models: applications and data analysis methods. Thousand Oaks: Sage.
Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357.
Schellens, T., Van Keer, H., & Valcke, M. (2005). The impact of role assignment on knowledge construction in asynchronous discussion groups: A multilevel analysis. Small Group Research, 36, 704–745.
Schellens, T., Van Keer, H., Valcke, M., & De Wever, B. (2007). Learning in asynchronous discussion groups: A multilevel approach to study the influence of student, group and task characteristics. Behaviour & Information Technology, 26, 55–71.
Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis. London: Sage.
Stevens, J. (1996). Applied multivariate statistics for the social sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
Strijbos, J. W., & Fischer, F. (2007). Methodological. challenges in collaborative learning research. Learning & Instruction, 17, 389–393.
Strijbos, J. W., Martens, R. L., Jochems, W. M. G., & Broers, N. J. (2004). The effect of functional roles on group efficiency: Using multilevel modeling and content analysis to investigate computer-supported collaboration in small groups. Small Group Research, 25, 195–229.
Strijbos, J. W., Martens, R. L., Jochems, W. M. G., & Broers, N. J. (2007). The effect of functional roles on perceived group efficiency during computer-supported collaborative learning: a matter of triangulation. Computers in Human Behavior, 23, 353–380.
Acknowledgments
The author would like to thank three anonymous reviewers for their helpful comments on an earlier version of this article.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Cress, U. The need for considering multilevel analysis in CSCL research—An appeal for the use of more advanced statistical methods. Computer Supported Learning 3, 69–84 (2008). https://doi.org/10.1007/s11412-007-9032-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11412-007-9032-2