Abstract
This study reports on a novel design methodology, namely, design-centered research (DCR), developed to analyze and evaluate the alignment between an online collaborative learning design and its enactment. The approach is illustrated in a study involving 40 groups in total. Twenty different online collaborative learning activities were designed and enacted by 20 groups of three students in each of two iterations. The collaborative learning design plans from the first round were adjusted after reflecting on misalignments observed through the method during the enactment, and then enacted and tested again by another 20 groups in the second round. The proposed method involves an interaction path graph as well as three proposed indicators of group functioning. These three indicators include: (a) the range of activated knowledge, (b) the degree of knowledge building, and (c) an interactivity of the approach. This approach to quantification of alignment between a collaborative learning design and its enactment was successful in revealing areas for improvement of the design. The results of the two round study indicate that the alignment significantly improved after the optimization of the collaborative learning design based on the analysis of the first round. The findings also suggest that optimizing a collaborative learning design using this method is associated with improvements in group performance. Building on these findings, the collaborative learning design framework is discussed in detail in this article, and resulting implications for practitioners are discussed in depth.
Similar content being viewed by others
References
Alharbi, N. M., Athauda, R. I., & Chiong, R. (2018). Empowering collaboration in project-based learning using a scripted environment: Lessons learned from analysing instructors’ needs. Technology, Pedagogy and Education, 27(3), 381–397. https://doi.org/10.1080/1475939X.2018.1473289.
Altebarmakian, M., & Alterman, R. (2019). Cohesion in online environments. International Journal of Computer-Supported Collaborative Learning, 14(4), 443–465. https://doi.org/10.1007/s11412-019-09309-y.
Anderson, T., & Shattuck, J. (2012). Design-based research: A decade of progress in education research? Educational Researcher, 41(1), 16–25. https://doi.org/10.3102/0013189x11428813.
Asensio-Pérez, J. I., Dimitriadis, Y., Pozzi, F., Hernández-Leo, D., Prieto, L. P., Persico, D., & Villagrá-Sobrino, S. L. (2017). Towards teaching as design: Exploring the interplay between full-lifecycle learning design tooling and teacher professional development. Computers & Education, 114, 92–116. https://doi.org/10.1016/j.compedu.2017.06.011.
Avcı, Ü. (2020). Examining the role of sentence openers, role assignment scaffolds and self-determination in collaborative knowledge building. Educational Technology Research and Development, 68(1), 109–135. https://doi.org/10.1007/s11423-019-09672-5.
Barab, S., & Squire, K. (2004). Design-based research: Putting a stake in the ground. The Journal of the Learning Sciences, 13(1), 1–14. https://doi.org/10.1207/s15327809jls1301_1.
Barron, B. (2003). When smart groups fail. The Journal of the Learning Sciences, 12(3), 307–359. https://doi.org/10.1207/S15327809JLS1203_1.
Bober, M. J., Sullivan, H. J., & Harrison, L. P. (1998). Instructional practices of teachers enrolled in educational technology and general education programs. Educational Technology Research and Development, 46(3), 81–97. https://doi.org/10.2307/30221066.
Burroughs, N., Gardner, J., Lee, Y., Guo, S., Touitou, I., Jansen, K., & Schmidt, W. (2019). Relationships between instructional alignment, time, instructional quality, teacher quality, and student mathematics achievement. In N. Burroughs et al. (Eds.), Teaching for excellence and equity (pp. 63–100). Cham: Springer.
Carter, L. (2008). Five big ideas: Leading total instructional alignment. Bloomington: Solution Tree Press.
Chen, B., Scardamalia, M., & Bereiter, C. (2015). Advancing knowledge-building discourse through judgments of promising ideas. International Journal of Computer-Supported Collaborative Learning, 10(4), 345–366. https://doi.org/10.1007/s11412-015-9225-z.
Cohen, S. (1987). Instructional alignment: Searching for a magic bullet. Educational Researcher, 16(8), 16–20. https://doi.org/10.2307/1175370.
Carvalho, L., Martinez-Maldonado, R., & Goodyear, P. (2019). Instrumental genesis in the design studio. International Journal of Computer-Supported Collaborative Learning, 14(1), 77–107. https://doi.org/10.1007/s11412-019-09294-2.
Demetriadis, S. N., Papadopoulos, P. M., Stamelos, I. G., & Fischer, F. (2008). The effect of scaffolding students’ context-generating cognitive activity in technology-enhanced case-based learning. Computers & Education, 51(2), 939–954. https://doi.org/10.1016/j.compedu.2007.09.012.
Design-Based Research Collective. (2003). Design-based research: An emerging paradigm for educational inquiry. Educational Researcher, 32(1), 5–8. https://doi.org/10.3102/0013189X032001005.
Dillenbourg, P., Baker, M., Blaye, A., & O’Malley, C. (1996). The evolution of research on collaborative learning. In E. Spada & P. Reiman (Eds.), Learning in humans and machine: Towards an interdisciplinary learning science (pp. 189–211). Oxford: Elsevier.
Dillenbourg, P. (1999). What do you mean by collaborative learning? In P. Dillenbourg (Ed.), Collaborative-learning: Cognitive and computational approaches (pp. 1–19). Oxford: Elsevier.
Dillenbourg, P., & Tchounikine, P. (2007). Flexibility in macro-scripts for computer-supported collaborative learning. Journal of Computer Assisted Learning, 23(1), 1–13. https://doi.org/10.1111/j.1365-2729.2007.00191.x.
Dillenbourg, P., Järvelä, S., & Fischer, F. (2009). The evolution of research on computer-supported collaborative learning. In N. Balacheff, S. Ludvigsen, T. D. Jong, A. Lazonder, & S. Barnes (Eds.), Technology-enhanced learning (pp. 3–19). Dordrecht: Springer.
De Wever, B., Schellens, T., Van Keer, H., & Valcke, M. (2008). Structuring asynchronous discussion groups by introducing roles: Do students act in line with assigned roles? Small Group Research, 39(6), 770–794. https://doi.org/10.1177/1046496408323227.
Easterday, M., Rees Lewis, D., & Gerber, E. (2014). Design-based research process: Problems, phases, and applications. In Proceeding of International Conference of Learning Sciences (pp. 1–8).
Engeström, Y. (1999). Activity theory and individual and social transformation. In Y. Engeström, R. Miettinen, & R.-L. Punamäki (Eds.), Perspectives on activity theory (pp. 19–38). New York, NY: Cambridge University Press.
Eryilmaz, E., van der Pol, J., Ryan, T., Clark, P. M., & Mary, J. (2013). Enhancing student knowledge acquisition from online learning conversations. International Journal of Computer-Supported Collaborative Learning, 8(1), 113–144. https://doi.org/10.1007/s11412-012-9163-y.
Eshuis, E. H., ter Vrugte, J., Anjewierden, A., Bollen, L., Sikken, J., & de Jong, T. (2019). Improving the quality of vocational students’ collaboration and knowledge acquisition through instruction and joint reflection. International Journal of Computer-Supported Collaborative Learning, 14(1), 53–76. https://doi.org/10.1007/s11412-019-09296-0.
Gerard, L., Kidron, A., & Linn, M. C. (2019). Guiding collaborative revision of science explanations. International Journal of Computer-Supported Collaborative Learning, 14(3), 291–324. https://doi.org/10.1007/s11412-019-09298-y.
Heimbuch, S., Ollesch, L., & Bodemer, D. (2018). Comparing effects of two collaboration scripts on learning activities for wiki-based environments. International Journal of Computer-Supported Collaborative Learning, 13(3), 331–357. https://doi.org/10.1007/s11412-018-9283-0.
Ingulfsen, L., Furberg, A., & Strømme, T. A. (2018). Students’ engagement with real-time graphs in CSCL settings: Scrutinizing the role of teacher support. International Journal of Computer-Supported Collaborative Learning, 13(4), 365–390. https://doi.org/10.1007/s11412-018-9290-1.
Johnson, C., Hill, L., Lock, J., Altowairiki, N., Ostrowski, C., dos Santos, L. D. R., & Liu, Y. (2017). Using design-based research to develop meaningful online discussions in undergraduate field experience courses. The International Review of Research in Open and Distributed Learning, 18(6), 36–53. https://doi.org/10.19173/irrodl.v18i6.2901.
Kapur, M., Voiklis, J., & Kinzer, C. K. (2011). A complexity-grounded model for the emergence of convergence in CSCL groups. In S. Puntambekar, G. Erkens, & C. Hmelo-Silver (Eds.), Analyzing interactions in CSCL (pp. 3–23). Boston, MA: Springer US.
Kramarski, B., & Dudai, V. (2009). Group-metacognitive support for online inquiry in mathematics with differential self-questioning. Journal of Educational Computing Research, 40(4), 377–404. https://doi.org/10.2190/EC.40.4.a.
Kreijns, K., Kirschner, P. A., & Jochems, W. (2003). Identifying the pitfalls for social interaction in computer-supported collaborative learning environments: A review of the research. Computers in Human Behavior, 19(3), 335–353.
Leinonen, T., Keune, A., Veermans, M., & Toikkanen, T. (2016). Mobile apps for reflection in learning: A design research in K-12 education. British Journal of Educational Technology, 47(1), 184–202. https://doi.org/10.1111/bjet.12224.
Lin, C. Y., & Reigeluth, C. M. (2016). Scaffolding wiki-supported collaborative learning for small-group projects and whole-class collaborative knowledge building. Journal of Computer Assisted Learning, 32(6), 529–547. https://doi.org/10.1111/jcal.12140.
Ludvigsen, S. (2016). CSCL towards the future: The second decade of ijCSCL. International Journal of Computer-Supported Collaborative Learning, 11(1), 1–7. https://doi.org/10.1007/s11412-016-9230-x.
Macphail, A., Tannehill, D., & Goc Karp, G. (2013). Preparing physical education preservice teachers to design instructionally aligned lessons through constructivist pedagogical practices. Teaching & Teacher Education, 33(2), 100–112. https://doi.org/10.1016/j.tate.2013.02.008.
Martin, F. (2011). Instructional design and the importance of instructional alignment. Community College Journal of Research and Practice, 35(12), 955–972. https://doi.org/10.1080/10668920802466483.
Martinez-Maldonado, R. (2019). A handheld classroom dashboard: Teachers’ perspectives on the use of real-time collaborative learning analytics. International Journal of Computer-Supported Collaborative Learning, 14(3), 383–411. https://doi.org/10.1007/s11412-019-09308-z.
Martone, A., & Sireci, S. G. (2009). Evaluating alignment between curriculum, assessment, and instruction. Review of Educational Research, 79(4), 1332–1361. https://doi.org/10.2307/40469099.
Meijer, H., Hoekstra, R., Brouwer, J., & Strijbos, J. W. (2020). Unfolding collaborative learning assessment literacy: A reflection on current assessment methods in higher education. Assessment & Evaluation in Higher Education, 1–19. https://doi.org/10.1080/02602938.2020.1729696.
Mende, S., Proske, A., Körndle, H., & Narciss, S. (2017). Who benefits from a low versus high guidance CSCL script and why? Instructional Science, 45(4), 439–468. https://doi.org/10.1007/s11251-017-9411-7.
Näykki, P., Isohätälä, J., Järvelä, S., Pöysä-Tarhonen, J., & Häkkinen, P. (2017). Facilitating socio-cognitive and socio-emotional monitoring in collaborative learning with a regulation macro script–an exploratory study. International Journal of Computer-Supported Collaborative Learning, 12(3), 251–279. https://doi.org/10.1007/s11412-017-9259-5.
Njenga, S. T., Oboko, R. O., Omwenga, E. I., & Muuro, E. M. (2017). Regulating group cognitive conflicts using intelligent agents in collaborative M-learning. In Proceedings of 2017 IEEE AFRICON (pp. 38–43). https://doi.org/10.1109/AFRCON.2017.8095452.
Pattalitan, J. A. (2016). The implications of learning theories to assessment and instructional scaffolding techniques. American Journal of Educational Research, 4(9), 695–700. https://doi.org/10.12691/education-4-9-9.
Popper, K. R. (1963). Science as falsification. Conjectures and refutations, 1, 33–39.
Raes, A., Schellens, T., De Wever, B., & Vanderhoven, E. (2012). Scaffolding information problem solving in web-based collaborative inquiry learning. Computers & Education, 59(1), 82–94. https://doi.org/10.1016/j.compedu.2011.11.010.
Rienties, B., Nguyen, Q., Holmes, W., & Reedy, K. (2017). A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK. Interaction Design and Architecture (s), 33, 134–154.
Roach, A. T., Niebling, B. C., & Kurz, A. (2008). Evaluating the alignment among curriculum, instruction, and assessment: Implications and applications for research and practice. Psychology in the Schools, 45(2), 158–176. https://doi.org/10.1002/pits.20282.
Rodríguez-Triana, M. J., Martínez-Monés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2015). Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations. British Journal of Educational Technology, 46(2), 330–343. https://doi.org/10.1111/bjet.12198.
Shin, Y., Kim, D., & Jung, J. (2018). The effects of representation tool (visible-annotation) types to support knowledge building in computer-supported collaborative learning. Journal of Educational Technology & Society, 21(2), 98–110.
Stahl, G., Koschmann, T., & Suthers, D. (2006). Computer-supported collaborative learning: An historical perspective. In R. K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 409–426). Cambridge: Cambridge University Press.
Stahl, G. (2017). Group practices: A new way of viewing CSCL. International Journal of Computer-Supported Collaborative Learning, 12(1), 113–126. https://doi.org/10.1007/s11412-017-9251-0.
Stahl, G., & Hakkarainen, K. (2019). Theories of CSCL. In U. Cress, C. Rosé, A. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning. New York: Springer http://GerryStahl.net/pub/cscltheories.pdf.
Tissenbaum, M., & Slotta, J. (2019). Supporting classroom orchestration with real-time feedback: A role for teacher dashboards and real-time agents. International Journal of Computer-Supported Collaborative Learning, 14(3), 325–351. https://doi.org/10.1007/s11412-019-09306-1.
Tversky, A. (1977). Features of similarity. Psychological Review, 84(4), 327–352. https://doi.org/10.1037/0033-295X.84.4.327.
Vogel, F., Wecker, C., Kollar, I., & Fischer, F. (2017). Socio-cognitive scaffolding with computer-supported collaboration scripts: A meta-analysis. Educational Psychology Review, 29, 477–511. https://doi.org/10.1007/s10648-016-9361-7.
Vygotsky, L. S. (1978). Mind in society. Cambridge: Harvard University Press.
Wang, X., & Mu, J. (2017). Flexible scripting to facilitate knowledge construction in computer-supported collaborative learning. Singapore: Springer.
Wake, J. D., Guribye, F., & Wasson, B. (2018). Learning through collaborative design of location-based games. International Journal of Computer-Supported Collaborative Learning, 13(2), 167–187. https://doi.org/10.1007/s11412-018-9278-x.
Webb, N. L. (1997). Criteria for alignment of expectations and assessments in mathematics and science education. Academic Achievement, 1(11), 1–46. https://doi.org/10.1111/j.1467-9892.1990.tb00038.x.
Yang, K. (2013). DBR and DCR, which can bridge the gap between educational theory and practice? E-Education Research, 12, 11–15. https://doi.org/10.13811/j.cnki.eer.2013.12.010.
Yang, K., & Liu, H. (2018). A case study on design of problem-solving collaborative learning from DCR perspective. E-Education Research, 11, 5–12.
Yao, Q., Yang, K., Zhao, G., & Huang, R. (2006). A concept map scoring algorithm based on proposition chains for concept mapping. In Proceeding of the second international conference on concept mapping (pp. 8–15). San Jose: Universidad de Costa Rica.
You, Y. (1993). What can we learn from chaos theory? An alternative approach to instructional systems design. Educational Technology Research and Development, 41(3), 17–32.
Zheng, B., Niiya, M., & Warschauer, M. (2015). Wikis and collaborative learning in higher education. Technology, Pedagogy and Education, 24(3), 357–374. https://doi.org/10.1080/1475939X.2014.948041.
Zheng, L. (2015a). A systematic literature review of design-based research from 2004 to 2013. Journal of Computers in Education, 2(4), 399–420. https://doi.org/10.1007/s40692-015-0036-z.
Zheng, L. (2015b). A case study on the consistence between the instructional design and the implementation of teaching. Modern Distance Education Research, 3, 95–103.
Zheng, L. (2017). Knowledge building and regulation in computer-supported collaborative learning. Singapore: Springer.
Zheng, L., Li, X., Zhang, X., & Sun, W. (2019). The effects of group metacognitive scaffolding on group metacognitive behaviors, group performance, and cognitive load in computer-supported collaborative learning. The Internet and Higher Education, 42, 13–24. https://doi.org/10.1016/j.iheduc.2019.03.002.
Zheng, L., & Yang, K. (2014). Why we should research on the consistency rather than the effectiveness? Chinese Educational Technology, 9, 20–23.
Zheng, L., Yang, K., & Huang, R. (2012). Analyzing interactions by an IIS-map-based method in face-to-face collaborative learning: An empirical study. Journal of Educational Technology & Society, 15(3), 116–132.
Acknowledgements
This study is funded by the youth project of Humanities and Social Science Research in the Ministry Education (19YJC880141) and National Natural Science Foundation of China (61907003).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Zheng, L., Cui, P. & Zhang, X. Does collaborative learning design align with enactment? An innovative method of evaluating the alignment in the CSCL context. Intern. J. Comput.-Support. Collab. Learn 15, 193–226 (2020). https://doi.org/10.1007/s11412-020-09320-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11412-020-09320-8