The need for considering multilevel analysis in CSCL research—An appeal for the use of more advanced statistical methods

Article

Abstract

Per definition, CSCL research deals with the data of individuals nested in groups, and the influence of a specific learning setting on the collaborative process of learning. Most well-established statistical methods are not able to analyze such nested data adequately. This article describes the problems which arise when standard methods are applied and introduces multilevel modelling (MLM) as an alternative and adequate statistical approach in CSCL research. MLM enables testing interactional effects of predictor variables varying within groups (for example, the activity of group members in a chat) and predictors varying between groups (for example, the group homogeneity created by group members’ prior knowledge). So it allows taking into account that an instruction, tool or learning environment has different but systematic effects on the members within the groups on the one hand and on the groups on the other hand. The underlying statistical model of MLM is described using an example from CSCL. Attention is drawn to the fact that MLM requires large sample sizes which are not provided in most CSCL research. A proposal is made for the use of some analyses which are useful.

Keywords

Multilevel models Hierarchical linear models Quantitative analysis for CSCL 

Notes

Acknowledgments

The author would like to thank three anonymous reviewers for their helpful comments on an earlier version of this article.

References

  1. Bonito, J. A. (2002). The analysis of participation in small groups: Methodological and conceptual issues related to interdependence. Small Group Research, 33, 412–438.CrossRefGoogle Scholar
  2. Bonito, J. A., & Lambert, B. L. (2005). Information similarity as a moderator of the effect of gender on participation in small groups: A multilevel analysis. Small Group Research, 36, 139–165.CrossRefGoogle Scholar
  3. Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models. Newbury Park, CA: Sage.Google Scholar
  4. Burstein, L. (1978). Assessing differences between grouped and individual-level regression coefficients. Sociological Methods & Research, 7, 5–28.CrossRefGoogle Scholar
  5. Burstein, L. (1980). The analysis of multilevel data in educational research and evaluation. Review of Research in Education, 8, 158–232.CrossRefGoogle Scholar
  6. Burstein, L., Kim, S. S., & Delandshere, G. (1989). Multilevel investigations of systematically varying slopes: Issues, alternatives, and consequences. In D. Bock (Ed.) Multilevel analysis of educational data (pp. 233–279). San Diego: Academic.Google Scholar
  7. Chiu, M. M., & Khoo, L. (2003). Rudeness and status effects during group problem solving: Do they bias evaluations and reduce the likelihood of correct solutions. Journal of Educational Psychology, 95, 506–523.CrossRefGoogle Scholar
  8. Chiu, M. M., & Khoo, L. (2005). A new method for analyzing sequential processes: Dynamic multilevel modeling. Small Group Research, 36, 600–631.CrossRefGoogle Scholar
  9. Cress, U. (2005). Ambivalent effect of member portraits in virtual groups. Journal of Computer-Assisted Learning, 21, 281–291.CrossRefGoogle Scholar
  10. De Wever, B., Van Keer, H., Schellens, T., & Valcke, M. (2007). Applying multilevel modelling to content analysis data: Methodological issues in the study of role assignment in asynchronous discussion groups. Leaning & Instruction, 17, 436–447.CrossRefGoogle Scholar
  11. Hox, J. J. (2002). Multilevel analysis: techniques and applications. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  12. Hron, A., Hesse, F. W., Cress, U., & Giovis, C. (2000). Implicit and explicit dialogue structuring in virtual learning groups. British Journal of Educational Psychology, 70, 53–64.CrossRefGoogle Scholar
  13. Kashy, D. A., & Kenny, D. A. (2000). The analysis of data from dyads and groups. In H. T. Reis, & C. M. Judd (Eds.) Handbook of research methods in social and personality psychology. Cambridge, UK: Cambridge University Press.Google Scholar
  14. Kenney, D. A., & Judd, C. M. (1986). Consequences of violating the independence assumption in analysis of variance. Psychological Bulletin, 99, 422–431.CrossRefGoogle Scholar
  15. Kenny, D. A., Kashy, D. A., & Bolger, N. (1998). Data analysis in social psychology. In D. T. Gilbert, S. T. Fiske, & G. Lindzey (Eds.) The handbook of social psychology (4th ed., Vol. 1, pp. 233–265). New York: McGraw Hill.Google Scholar
  16. Kenny, D. A., Mannetti, L., Pierro, A., Livi, S., & Kashy, D. A. (2002). The statistical analysis of data from small groups. Journal of Personality and Social Psychology, 83, 126–137.CrossRefGoogle Scholar
  17. Kimmerle, J. & Cress (2008). Group Awareness and Self-Presentation in Computer-Supported Information Exchange. International Journal of Computer-Supported Collaborative Learning DOI 10.1007/s11412-007-9027-z.
  18. Kreft, I. (1996). Are multilevel techniques necessary? An overview, including simulation studies. Retrieved March, 20, 2007 from http://www.calstatela.edu/faculty/ikreft/quarterly/quarterly.html.
  19. Maas, C. J. M., & Hox, J. (2005). Sufficient samples sizes for multilevel modeling. Methodology, 1, 86–92.CrossRefGoogle Scholar
  20. Marsh, H. W., & Hau, K. T. (2003). Big-fish-little-pond effect on academic self-concept: A cross-cultural (26-Country) test of the negative effects of academically selective schools. American Psychologist, 58(5), 364–376.CrossRefGoogle Scholar
  21. McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.CrossRefGoogle Scholar
  22. Paccagnella, O. (2006). Centering or not centering in multilevel models? The role of the group mean and the assessment of group effects. Evaluation Review, 30, 66–85.CrossRefGoogle Scholar
  23. Piontkowski, U., Keil, W., & Hartmann, J. (2006). Analyseebenen und Dateninterdependenz in der Kleingruppenforschung am Beispiel netzbasierter Wissensintegration. Zeitschrift für Sozialpsychologie, 37, 41–50.CrossRefGoogle Scholar
  24. Raudenbush, S., & Bryk, A. (2002). Hierarchical linear models: applications and data analysis methods. Thousand Oaks: Sage.Google Scholar
  25. Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357.CrossRefGoogle Scholar
  26. Schellens, T., Van Keer, H., & Valcke, M. (2005). The impact of role assignment on knowledge construction in asynchronous discussion groups: A multilevel analysis. Small Group Research, 36, 704–745.CrossRefGoogle Scholar
  27. Schellens, T., Van Keer, H., Valcke, M., & De Wever, B. (2007). Learning in asynchronous discussion groups: A multilevel approach to study the influence of student, group and task characteristics. Behaviour & Information Technology, 26, 55–71.CrossRefGoogle Scholar
  28. Snijders, T. A. B., & Bosker, R. J. (1999). Multilevel analysis. London: Sage.Google Scholar
  29. Stevens, J. (1996). Applied multivariate statistics for the social sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.Google Scholar
  30. Strijbos, J. W., & Fischer, F. (2007). Methodological. challenges in collaborative learning research. Learning & Instruction, 17, 389–393.CrossRefGoogle Scholar
  31. Strijbos, J. W., Martens, R. L., Jochems, W. M. G., & Broers, N. J. (2004). The effect of functional roles on group efficiency: Using multilevel modeling and content analysis to investigate computer-supported collaboration in small groups. Small Group Research, 25, 195–229.CrossRefGoogle Scholar
  32. Strijbos, J. W., Martens, R. L., Jochems, W. M. G., & Broers, N. J. (2007). The effect of functional roles on perceived group efficiency during computer-supported collaborative learning: a matter of triangulation. Computers in Human Behavior, 23, 353–380.CrossRefGoogle Scholar

Copyright information

© International Society of the Learning Sciences, Inc.; Springer Science+ Business Media, LLC 2008

Authors and Affiliations

  1. 1.Knowledge Media Research CenterTuebingerGermany

Personalised recommendations