Introduction

During collaborative learning, learners engage in collaborative activities that are expected to facilitate deep elaboration of the learning material, such as explaining, argumentation, or identifying and negotiating socio-cognitive conflicts (King 2007). Engaging in such activities should enhance learners’ acquisition of knowledge and skills (Chi 2009). However, without guidance, learners often have difficulties engaging in the expected collaborative activities, and thus fail to take advantage of collaborative learning opportunities. For instance, learners do not engage in higher-level argumentation (Bell 2004) or do not take unshared information into account (Stasser and Titus 1985). As an explanation of why collaborative learning falls short of expectations, King (2007) argues that learners might have a limited conception of how to interact with each other. In particular, interacting with others in learning environments based on educational technology, such as MOOCS or computer-mediated interactions among remote learners, creates difficulties for learners unfamiliar with these new learning settings (Fischer et al. 2013).

However, while educational technology requires learners to engage in unfamiliar collaborative learning settings, it also makes it possible to shape and scaffold collaborative learning (Dillenbourg et al. 2009). For example, scripts for computer-supported collaborative learning (CSCL scripts) have been designed to provide just-in-time scaffolds to structure and sequence collaborative learning activities. CSCL scripts repeatedly engage learners in beneficial collaborative practices, eventually supporting the development of collaboration skills and domain learning (Fischer et al. 2013). However, studies on the effectiveness of CSCL scripts show heterogeneous results for domain learning and collaboration skills. In the light of the relevance of guidance for collaborative learning in computer-supported learning environments such as MOOCs and blended learning environments, and the diversity and complexity of previous findings, summarizing efforts seem warranted.

In a qualitative approach to summarizing CSCL research, Wise and Schwarz (2017) collected widespread views of CSCL among experts in the field and developed them into thought-provoking positions. One result of this effort holds that CSCL scripts are prone to overscripting by including prescriptions for users’ interactions that are too specific. This leads to a high risk of undermining learners’ self-determination and agency and thus reduces intrinsic forms of motivation to fully engage in collaborative learning activities (Deci and Ryan 1985). They further argue that positive effects, if they exist at all, are restricted to the enhancement of collaboration skills. In a recently conducted quantitative approach, a meta-analysis of the effects of CSCL scripts found a significant small effect of CSCL scripts on domain learning and a significant large effect on collaboration skills (Vogel et al. 2017). These results partially support the argument that CSCL scripts mainly affect the learning of collaboration skills; however, they also show that despite possible overscripting effects, CSCL scripts can be effective for domain learning as well. Yet it is still an open question whether the effectiveness of CSCL scripts is decreased by the hypothesized negative influence on motivation (Dillenbourg 2002).

Apart from this, however, the recent meta-analysis on the effects of CSCL scripts did not sufficiently clarify how and under which circumstances CSCL scripts are effective for learning, particularly for domain learning. Although some moderators were tested, a significant amount of variance remained, leaving the question of what makes CSCL scripts particularly effective mostly unanswered. Moreover, several new studies on CSCL scripts have been published since this meta-analysis was conducted in 2013. Thus, a quantitative analysis of the effects of CSCL scripts beyond their influence on domain learning and collaboration skills seems necessary at this stage. Thus, the present meta-analysis contributes to answering the question of whether CSCL scripts negatively influence learners’ motivation, whether the effects of CSCL scripts on domain learning and collaboration skills remain robust when integrating more recent CSCL script studies, and to what extent specific features of CSCL scripts explain their effectiveness.

Scaffolding collaboration using CSCL scripts

In their script theory of guidance, Fischer et al. (2013) argue that collaborative learners’ failure to engage in high-level collaborative processes is an indicator of a lack of internal collaboration scripts. Internal collaboration scripts are flexible cognitive structures consisting of knowledge about specific collaborative practices. In line with dynamic memory theory (Schank 1999), internal collaboration scripts consist of knowledge components about situations that are flexibly stored in a person’s memory and activated in a more or less likely sequence depending on a person’s goals and situational characteristics. This general knowledge structure can be dynamically changed, allowing for spontaneous reactions to situational changes (Schank 1999). If learners do not have well-functioning internal collaboration scripts for a specific learning situation, they will engage in processes that are less beneficial for learning, and hence not take full advantage of the given learning environment. To support learners in overcoming dysfunctional internal collaboration scripts, the information needed to engage in beneficial collaborative learning processes can be provided externally. For this purpose, CSCL scripts scaffold learners towards collaboration in a specific context by affording specific activities (e.g., argumentation) in a likely sequence (e.g., first reading a text, then formulating arguments based on the text), and implicitly or explicitly distributing roles among learners (e.g., pro and con positions). That means that CSCL scripts can be understood as scaffolding for the social interactions necessary for collaborative learning (Kollar et al. 2006).

Like other types of scaffolds, CSCL scripts support learners in solving tasks they would not be able to solve without the scaffold (Wood et al. 1976). Therefore, CSCL scripts are expected to be most effective when they fit the learners’ prior knowledge and skills (i.e., the internal script they have available, Fischer et al. 2013). What distinguishes CSCL scripts from other types of scaffolds is the particular knowledge and skills targeted by the different scaffolds. Other scaffolds such as self-explanation prompts provide help on a content-related level to support the development of individual problem-solving processes in a specific domain (e.g., Heitzmann et al. 2015). Such scaffolds address conceptual, metacognitive, strategic, or motivational aspects of individual problem-solving (Belland et al. 2017). They particularly target the development of individual problem-solving skills to enable learners to solve similar problems in the specified domain on their own. In contrast, CSCL scripts are scaffolds that support collaborative learning and collaborative problem-solving processes such as coordination, sharing of information and ideas, or negotiation (Liu et al. 2015). Instead of directly addressing the content specific to the learning domain, CSCL scripts mainly target the development of skills necessary to interact in a social learning setting or to solve problems collaboratively. By that, CSCL scripts indirectly facilitate domain learning by engaging learners in beneficial activities (King 2007). Hence, CSCL scripts facilitate both collaboration skills and domain specific knowledge and skills.

To explain the organization of CSCL scripts, Fischer et al. (2013) identify four hierarchical script components: play, scene, scriptlet, and role, with the play component representing the highest level. The play component encompasses knowledge about the general task (e.g., argumentation), including knowledge about each individual’s role and the sequence of scenes. The scene component consists of knowledge about situations making up a given scene (e.g., formulating a counterargument to an argument that has been stated). The scriptlet component consists of knowledge about activities and their sequence within a specific scene (e.g., when formulating an argument, first develop a claim, then support the claim with data and warrants). Finally, the role component consists of knowledge about participants’ specific roles during a scene (e.g., participants in an argumentation with different positions). External CSCL scripts can provide scaffolding at all four script component levels. With their optimal external scripting level principle, Fischer et al. (2013) address the overscripting effect by identifying a mechanism to counteract overscripting. The principle suggests that learners benefit most from external CSCL scripts that address the hierarchical level whose subordinate components are already available to the learners. For example, if a learner already knows how to formulate a sound argument including data and warrants, but doesn’t know how to respond to a collaboration partner’s argument with a counterargument, the CSCL script should prompt the learner how to produce counterarguments. Taking the optimal external scripting level into account, a CSCL script could lead to overscripting if it addresses internal script components on a level the learners are already able to activate. For instance, if the learners already know that raising an argument involves formulating a claim, reasons and warrants, then a detailed prompt to formulate these components of an argument would not be optimal. In this case, the CSCL script might reduce learners’ autonomy, decreasing their motivation and distracting them from using the functional internal script components they already have available (Dillenbourg 2002; Fischer et al. 2013).

In sum, collaborative learning requires complex skills, which are cognitively represented in internal scripts learners partially share socially. Often, it seems that learners do not have a functional internal script for a specific situation available. This leads to the use of less functional collaborative activities in collaborative learning situations. Hence, learners fail to engage in high-quality collaboration and thus do not take full advantage of the collaborative learning situation. External CSCL scripts are an instructional means that may compensate for erroneously activated internal script components or a lack of internal collaboration scripts and thus help learners engage in social interactions that lead to the enhancement of both domain learning and collaboration skills. CSCL scripts are specified by the target of social processes and can be distinguished from other scaffolds that target content-related processes in the learning domain.

Effects of CSCL scripts on motivation

External CSCL scripts have been criticized for constraining learners during collaboration and thereby undermining learners’ motivation (Dillenbourg 2002). Motivation comprises all factors that are important for the selection, initiation, and maintenance of actions (e.g., Heckhausen 1974) and is hence highly important for individual learning (Urhahne 2008). The critique mentioned above is related to phenomena best explained by self-determination theory (SDT). In SDT, Deci and Ryan (1985) describe the experience of basic psychological needs as a key determinant of human behavior and emphasize the importance of the social environment for individual motivation (Ryan and Deci 2008b). The basic psychological needs are autonomy, competence, and relatedness. Autonomy refers to whether a specific behavior exhibited by an individual is perceived as congruent and volitional. Competence concerns the perceived efficacy of one’s behavior. Relatedness refers to one’s personal feeling of connection with a specific community and the significance thereof (Ryan and Deci 2008a). These three basic psychological needs are seen “as a nutrient essential for psychological growth, integrity, and wellness” (Ryan and Deci 2008a, p. 657). These needs are considered universal, and humans naturally seek to fulfill them by exhibiting or avoiding certain behaviors. This theory has received particular attention in educational contexts (e.g., Krapp et al. 2014). It is assumed that learners are autonomously motivated and thus engage in learning activities in learning environments that promote autonomy, perceptions of competence, and social relatedness. In contrast, learners experience controlled motivation or even amotivation and thus only superficially engage in learning activities if their basic psychological needs are not appropriately addressed (Ryan and Deci 2008a).

With respect to external CSCL scripts, this leads to two conceivable scenarios: On the one hand, the highly coercive nature of external CSCL scripts might diminish learners’ perceived autonomy during the collaboration process and thus reduce motivation to learn collaboratively (Dillenbourg 2002). On the other hand, external CSCL scripts might enhance learners’ perceived competence by enabling them to experience success early on. This might increase learners’ motivation to stay engaged in the collaborative learning situation, reduce negative effects of unequal participation (i.e., social loafing and sucker effects; Latané et al. 1979; Schnake 1991) and thus further enhance motivation to learn collaboratively (Weinberger et al. 2009). While the most prominent articles criticizing CSCL scripts for undermining learners’ motivation are conceptual (Dillenbourg 2002; Wise and Schwarz 2017), only a few studies have examined the effect of CSCL scripts on motivation empirically. For example, Peterson and Roseth (2016) found a non-significant negative effect on motivation when comparing learners in a synchronous CSCL environment learning with a CSCL script to those learning without a CSCL script. However, in the same study, a positive effect of a CSCL script on motivation was found for students in an asynchronous learning environment. In a similar vein, Demetriadis et al. (2011) found that learners supported by an external peer-review script experienced higher motivation from the tasks and the collaboration itself and were less motivated by grading than learners who were not supported by an external CSCL script.

To summarize, SDT suggests that in order for learners to be autonomously motivated, the learning environment should support their perceptions of autonomy, competence, and relatedness. Although external CSCL scripts have been criticized for hampering learners’ motivation (e.g., Wise and Schwarz 2017), research examining how external CSCL scripts influence basic psychological needs and, thus also, motivation is still inconclusive. The present meta-analysis addresses this ambiguity by quantitatively synthesizing existing findings on the effect of CSCL scripts on motivational outcomes for the first time.

Effects of CSCL scripts on domain learning and collaboration skills

Several studies examining the effects of CSCL scripts on learning have been published in the last two decades (e.g., Choi et al. 2005; Tsovaltzi et al. 2015). Past research syntheses on CSCL scripts have shown that they have significantly higher effects on learners’ collaboration skills than on domain learning (Vogel et al. 2017). A reason for this difference in effect sizes might be related to the cognitive resources available for the internalization of collaborative activities and the elaboration of content-related knowledge and skills based on these activities. During beneficial social interactions, learners co-construct knowledge that is internalized and integrated by each individual learner (Teasley 1997; Weinberger et al. 2007). The most beneficial interactions typically include higher cognitive processes (Chi and Wylie 2014). Collaboration skills are internalized by the repeated practice and engagement in collaborative activities scaffolded by CSCL scripts, while domain learning is enhanced by the cognitive elaboration induced by the collaborative activities a CSCL script requires the learners to engage in. Thus, learners’ limited cognitive resources must be divided among both the internalization of collaboration skills and the elaboration of domain knowledge based on these activities. Learners with less elaborate internal collaboration scripts who are scaffolded with external CSCL scripts might still need to put a higher amount of cognitive effort into the collaborative task, leaving fewer cognitive resources available for the elaboration of domain knowledge (Kirschner et al. 2018). This might lead to higher effects of CSCL scripts on collaboration skills than on domain learning. Overall, CSCL scripts seem to positively affect domain learning and the learning of collaboration skills. It seems plausible to assume that the effect of CSCL scripts on domain learning depends on how functional a learner’s internal collaboration scripts are.

Effectiveness of different collaborative activities scaffolded by CSCL scripts

Although the meta-analysis from Vogel et al. (2017) yielded important insights to the extent to which CSCL scripts affect learning, it only allows for limited conclusions on the question of how CSCL scripts affect learning. On a general level, it seems plausible to assume that CSCL scripts foster domain learning and collaboration skills by engaging learners in functional collaborative activities. Learners internalize the knowledge and skills needed to perform such collaborative activities by developing or modifying their internal collaboration scripts (Fischer et al. 2013). These collaborative activities (e.g., negotiation) are typically associated with higher-order cognitive processes that are beneficial for domain learning (Chi and Wylie 2014; King 2007). Vogel et al.’s (2017) meta-analysis approached the question of how CSCL scripts affect learning by analyzing whether scripts asking learners to engage in transactive activities are more effective than scripts asking learners to engage only in activities that are not transactive. During transactive activities, two or more learners elaborate on each other’s ideas. In the best case, transactive activities result in new outcomes that neither learner would have come up with alone (Teasley 1997). Based on the ICAP framework (Chi and Wylie 2014) the collaborative activities expected to be most beneficial for learning are those in which learners mutually build on each other’s contributions while cognitively processing the given learning material(Teasley et al. 2008). Such transactive activities should lead to a deeper elaboration of the learning material and hence higher domain learning gains (Teasley 1997). Vogel et al.’s (2017) meta-analysis compared CSCL scripts that did and did not scaffold transactive activities during collaboration. Their descriptive results indicated significant positive effects only for CSCL scripts scaffolding transactive activities on domain learning, but detrimental effects on collaboration skills. The differences between the effects of CSCL scripts scaffolding and not scaffolding transactive activities were not significant and left a substantial amount of unexplained variance. Since transactivity encompasses a large number of more specific collaborative activities (e.g., answering questions, expressing critiques, synthesizing different arguments, completing others’ ideas), it seems to be reasonable to analyze more thoroughly how scaffolding more specific collaborative activities might explain the effectiveness of CSCL scripts. In doing so, we apply the collaborative problem-solving framework of Liu et al. (2015), which describes three different social activities that are needed for successful collaborative problem solving. CSCL scripts can support learners in engaging in these social activities, thus enhancing both domain learning and the learning of collaboration skills.

For collaborative learning to be successful, learners must demonstrate, alongside cognitive competences such as task-specific and domain-specific problem-solving skills, the skills to engage in social activities (e.g., Tschan et al. 2009; Zhuang et al. 2008). In their collaborative problem solving framework, Liu et al. (2015) propose three different social activities that are beneficial for learning: sharing ideas or information, negotiating, and regulating or coordinating problem solving. Sharing ideas or information refers to the skill needed for exchanging unshared information or ideas in a way that is appropriate for the collaboration partners. Information sharing is assumed to be beneficial for constructing and maintaining a shared representation and understanding of the problem (Roschelle and Teasley 1995) as well as for better decision making and problem solving (Lu et al. 2012). Negotiating refers to the learner’s ability to criticize, elaborate on, and question their learning partners’ contributions (Liu et al. 2015). This type of collaborative activity has been shown to particularly enhance individual understanding and learning (Chi and Wylie 2014; Noroozi et al. 2013c; Teasley 1997). Coordinating problem solving includes meta-cognitive activities such as goal setting, evaluating strategies, or regulating motivation and emotions (Liu et al. 2015). According to Järvelä and Hadwin (2013), coordination of collaborative activities is essential for efficient, effective, and enjoyable learning. It is worth mentioning that this framework refers to collaborative problem solving in general without explicitly mentioning computer-supported collaboration. However, collaborative problem-solving approaches are increasingly used in CSCL research as they are more general and, thus suitable for systematic reviews. Furthermore, it is assumed that the same skills apply to learning in CSCL environments. However, learners must additionally be able to handle and coordinate the technological component of learning in CSCL. In some CSCL environments, learners communicate using communication technology such as audio or video conferences or an asynchronous chat program (e.g., a forum). If the discourse is limited due to missing eye contact or lack of opportunities to interpret facial expressions and gestures, more explicit communication strategies such as turn-taking must be used to maintain successful collaboration (Rummel et al. 2009).

In short, the collaborative activities described here seem essential for collaborative learning, and successful collaborative learners ought to have corresponding “functional configuration of internal script components” (Fischer et al. 2013, p. 61). Providing learners with external CSCL scripts allows them to overcome less functional configurations of internal collaboration scripts (Fischer et al. 2013). Consequently, the stimulation of specific collaborative activities using CSCL scripts can be considered a crucial mechanism for the effectiveness of CSCL scripts in facilitating learning. Therefore, we seek to systematically examine to what extent stimulating these collaborative activities explains effects of CSCL scripts on domain learning and the enhancement of collaboration skills.

Effectiveness of combining scaffolds for different types of collaborative activities

The application of all of the aforementioned collaborative activities are perceived to be beneficial for domain learning. Learners who engage in all of these collaborative activities during a collaborative problem-solving task exhibit a higher task performance compared to learners who engage in only some or none of them (Andrews-Todd and Forsyth 2018). Assuming that learners lack corresponding functional internal scripts, it seems to be reasonable to design CSCL scripts with specific scaffolding for each of these collaborative activities. Hence, the development of the corresponding collaboration skills as well as domain learning are needs that are addressed by CSCL scripts.

Tabak (2004) suggests a framework for the distribution of different scaffolds for one need or for different needs. She describes the combination of different scaffolds targeting different needs as differentiated scaffolding, while different scaffolds targeting the same need can also be combined deploying their maximal strengths in the form of synergistic scaffolding. Applying Tabak’s (2004) arguments to scaffolding different collaborative activities with CSCL scripts, differentiated and synergistic scaffolding could be relevant. For differentiated scaffolding, the proximate needs that are scaffolded using Liu et al.’s (2015) framework are the specific corresponding collaboration skills. Here, a scaffold targeting one collaborative activity exclusively meets the need to enhance the corresponding collaboration skill and not others. Thus, the effects of a CSCL script combining scaffolds for different collaborative activities should be as high as when the scaffolds were individually applied. Also, these scaffolds should not negatively interfere with each other (Tabak 2004).

However, scaffolding the different collaborative activities also addresses one joint need, namely domain learning. For domain learning, combining different scaffolds would ideally lead to synergistic scaffolding, as the scaffolds mutually increase each other’s effectiveness (Tabak 2004). In the case of synergistic scaffolding, the effect of combined scaffolds on domain learning would thus exceed the effects of CSCL scripts scaffolding each collaborative activity individually. However, existing experimental studies with CSCL scripts combining different scaffolds indicate that synergistic scaffolding is hard to achieve and the combination of scaffolds can even decrease their effectiveness (Kollar et al. 2014; Noroozi et al. 2013b ). A potential reason for the loss in effectiveness may be that combinations of different scaffolds can be overwhelming, particularly for weak learners (Schwaighofer et al. 2017). However, these studies focused on combining different CSCL scripts or combining CSCL scripts with other types of scaffolds. It is unclear whether these results are transferable to different types of activity prompts within a single CSCL script. Thus, it remains an open question whether CSCL scripts should scaffold different collaborative activities in a combined way within the same learning situation or scaffold one activity at a time in order to achieve the greatest benefits for domain learning. We assume that collaboration skills are acquired by repeatedly engaging in collaborative activities for practice, as described above (Fischer et al. 2013). In light of this, both the individual as well as the combined scaffolding of collaborative activities could be beneficial for learning. On the one hand, learners might benefit from combined scripting for different collaborative activities, as several skills necessary for successful collaboration are addressed simultaneously (Tabak 2004). On the other hand, learners might need to deliberately practice a single collaborative activity, particularly when the intervention period is short. Therefore, the meta-analysis at hand seeks to clarify to what extent scaffolding collaborative activities separately or in combination lead to better effects of CSCL scripts on domain learning and collaboration skills.

Goals and research questions

The first goal of this meta-analysis is to meta-analytically address the hypothesis that CSCL scripts are prone to overscripting and hence to reducing learners’ motivation. The second goal is to assess whether the positive effects of learning with CSCL scripts on domain learning and collaboration skills found by Vogel et al. (2017) hold when examining an updated and extended sample of primary studies on learning with CSCL scripts. In addition, we seek to distinguish between the effects of CSCL scripts on different outcomes related to domain learning and collaboration skills. The third goal is to test the proposed mechanisms for how CSCL scripts facilitate domain learning by stimulating collaborative activities to support the internalization of functional internal script components. Particularly, we are interested in how stimulating specific types of collaboration skills exclusively and in combination might differentially affect the effectiveness of CSCL scripts on domain learning. Based on the literature presented above, we propose the following research questions.

RQ 1: What is the overall effect of collaborative learning with CSCL scripts compared to unstructured collaborative learning on motivation?

RQ 2: What is the overall effect of collaborative learning with CSCL scripts compared to unstructured collaborative learning on domain learning and collaboration skills?

RQ 3: To what extent do the effects of CSCL scripts differ if they prompt one, two, or three different types of collaborative activities?

RQ 4: To what extent do the effects of CSCL scripts on domain learning and collaboration skills differ between CSCL scripts prompting a) negotiation, b) information sharing, and c) coordination individually, and in combination with each other?

Method

Criteria for inclusion

To be included in the present meta-analysis on the effects of CSCL scripts on motivation, domain learning, and collaboration skills, primary studies had to meet the following criteria:

Independent variable (1): Only (quasi-)experimental studies with scaffolding collaborative learning by means of a CSCL script as an independent variable (i.e., comparing groups of learners scaffolded with a CSCL script to groups of learners learning without a CSCL script) were included in the meta-analysis. More specifically, CSCL scripts as an instructional means were defined as follows. The scaffolding of a CSCL script must target the collaborative process of at least two learners. Additionally, at least one activity scaffolded by the CSCL script should ask learners to address their learning partner(s). Typically, a CSCL script supports the collaborative processes by structuring the collaborative task into sub-tasks, for example, by prompting specific activities in a specific order. In these sub-tasks, learners address their learning partners by, for instance, explaining, discussing, or asking questions. Some CSCL scripts also assign roles to the participants. However, scaffolds that only implicitly assigned roles to learners (for example, by distributing learning materials among students to induce knowledge interdependence) and did not include any further scaffolds targeting the collaborative process, were not defined as CSCL scripts (e.g., Molinari et al. 2009). Comparisons of two different CSCL scripts (e.g., Rummel et al. 2009), comparisons of CSCL scripts with other instructional means such as example-based learning (e.g., Rummel and Spada 2005), with individual learning (e.g., Peterson and Roseth 2016), or with waiting conditions were not considered in this meta-analysis. Hence, the only difference between the experimental group and the control group is the provision of the CSCL script in the experimental group.

Dependent variable (2): The present meta-analysis seeks to identify the effect of CSCL scripts on motivation and learning outcomes, specifically domain learning and collaboration skills. Therefore, only studies reporting at least one of these dependent variables were included in the analysis. We are primarily interested in the effect of CSCL scripts on motivation. Self-report questionnaires that assessed motivational factors (e.g., motivation to participate) based on self-determination theory or expectancy-value models were categorized as motivation. If several subscales for motivation were reported, all subscales were included but treated as statistically dependent. Negatively framed subscales (e.g., tension and pressure subscale in the Intrinsic Motivation Inventory) were inverted before being included in the analyses. We defined domain learning as knowledge and skills in the subject domain students learned about during the intervention. For example, if learners analyzed student cases based on attribution theory during the intervention, tests assessing knowledge and application of attribution theory were categorized as domain learning (e.g., Stegmann et al. 2007). We defined collaboration skills as knowledge and skills that are needed to engage in collaborative activities. For example, argumentation or knowledge about argumentation, as well as knowledge about the general procedure of collaboration were categorized as collaboration skills. All dependent variables were required to be assessed after and independently of the treatment and could be assessed either on the individual or on the small group level. Hence, performance measures assessed during the intervention were not included in the analyses.

Context of the study (3): Only studies using computer-supported learning settings with groups of at least two learners were included in the meta-analysis. Within this, technological means could be used to facilitate learners’ communication, the learners’ documentation of learning process and/or learning outcome, as well as the CSCL script could be provided by technological means.

Study design (4): Only studies that reported data from an experimental or a quasi-experimental design were included in the meta-analysis, i.e., studies needed to compare a treatment condition supported by CSCL scripts with a control condition not supported by CSCL scripts.

Availability of data (5): Only studies that reported enough data to calculate an effect size were included in this meta-analysis. To calculate Hedges’ g, information on the mean, standard deviation, and number of participants in the experimental group and the control group are needed. Alternatively, F, t, r, or χ2 statistics can be used to estimate Hedges’ g, if the size of both groups is known. However, before excluding studies reporting insufficient data, the authors were contacted in order to obtain missing data.

Article quality (6): Only articles that were published in English and in peer-reviewed journals were included in the meta-analysis. Publication in peer-reviewed articles is seen as an indication of quality. Additionally, published articles are accessible to a broad scientific community, which enables fellow researchers to replicate meta-analytic findings. Furthermore, articles published in conference proceedings were excluded as they often are a first version of articles later published in peer-reviewed journals. To avoid the inclusion of unidentifiable duplicates, these articles were not considered in the meta-analysis.

Literature search

Two comprehensive literature searches were conducted in April 2017 (first literature search) and in January 2020 (second literature search). For each literature search, we searched the databases ERIC and ISI Web of Science using the search terms “(scaffold* OR script*) AND (learn* OR know*) AND (collaborat* OR cooperat*) AND (computer* OR CSCL OR techno*)” and used a snowball system.

During the first literature search, the coding for inclusion was conducted in two steps for both the database search and the snowball system. The first step was based on the title and abstract of the article, while the second step was based on the full text. The database search resulted in 624 articles, excluding duplicates. As the search and inclusion criteria are based on the work of Vogel et al. (2017), all studies that were excluded during the literature search of Vogel et al. (2017), were also excluded from the present meta-analysis. This step left a total of 418 new studies. Then, based on a coding scheme, the titles and abstracts of all articles were coded for eligibility based on the aforementioned inclusion criteria 1, 3, and 4. The articles were coded as relevant or irrelevant by two independent coders, with double coding for a 10 % sample (Cohen’s Kappa = .86). In the second step, the full texts of the 71 articles coded as relevant in the first step were obtained and assessed for eligibility based on inclusion criteria 1 to 6. Two of the authors double-coded the full texts independently and discussed any disagreements, ultimately resulting in 32 eligible articles. For the snowball system, the reference lists of eligible articles published in 2015 or later were scanned for the search terms to identify additional articles that were not found by the database search. This procedure resulted in 34 articles, which were then obtained as full texts and coded for inclusion by both coders independently. Nine additional articles were found to be eligible by both coders (Cohen’s Kappa of the full text coding was 0.60). In total, the literature search in 2017 resulted in 41 articles.

For the second literature search in January 2020, the same key words and databases were used to identify articles that were published in 2017 and later. The second literature search yielded 202 articles (including 7 systematic reviews), for which the titles and abstracts were coded based on inclusion criteria 1, 3, and 4. Additionally, we screened the full texts and reference lists of the seven newly identified systematic reviews for additional relevant studies that had not been detected by the literature search. The database search and the snowball system resulted in a total of 76 articles (44 from the databases and 32 from the snowball system) that were included in the full text coding and coded for eligibility based on inclusion criteria 1 to 6. Of these, 12 articles were found to be eligible for the meta-analysis (9 from the databases and 3 from the snowball system). In the second literature search, all coding was independently conducted by two coders. All disagreements were discussed until agreement was reached.

Description of sample

In total, the literature search resulted in 52 articles reporting 56 relevant studies. Three studies were excluded after data extraction, as they reported duplicate data from studies that were also included in the sample (Bollen et al. 2015; Mäkitalo et al. 2005; Noroozi et al. 2013a). The total sample includes 49 articles reporting 53 relevant studies involving 5616 participants (M = 102.3, SD = 118.5). The articles were published between 2000 and 2020. The participants’ age ranged from 6 to 64. Most studies (k = 36) analyzed participants enrolled in higher education, 16 studies analyzed participants enrolled in primary or secondary school, and one study analyzed participants enrolled in professional training. In most studies, participants learned content related to the natural sciences (k = 17) followed by the social sciences (k = 16). Other subjects were the humanities (k = 4), medical science (k = 3), computer science (k = 7), and other topics (k = 6).

Coding of studies

A coding scheme describing the outcome variables and their specific components, as well as the type of collaborative activity (i.e., scripting for negotiation, information sharing, and coordination) was developed. The coding scheme contained definitions of the relevant variables, detailed descriptions of the coding rules and the coding procedure, and examples for each variable and its categories. The CSCL script for each study was categorized for each type of collaborative activity according to whether or not it included prompts for that specific collaborative activity. Two of the authors double coded 30% of the articles resulting from the first literature search. The same two persons coded half of the remaining studies each. For all variables, the coding was cross-checked by the respective second coder and differences were discussed in order to reach consensus. Following the second literature search, all studies found to be eligible for inclusion were double coded, and disagreements were discussed until consensus was reached. During the coding of the articles resulting from the second literature search, we refined the descriptions of the collaborative activities in the coding scheme. Hence, all codes from the first literature search were screened again by two coders in order to incorporate the refinements. The individual effects and categorization of each study regarding the moderator levels and dependent variables are reported in Table A4. For a summary of the co-occurrence of different moderator levels and dependent variables within studies, see Table A5.

Dependent variable: For the outcome variable domain learning, we distinguished between knowledge assessed in recall tests and application tests. Recall tests typically ask learners to reproduce knowledge that was learned beforehand without making inferences themselves. This knowledge typically consists of facts, definitions, or theories. Examples of recall tests are multiple choice tests and recognitions tests. In contrast, application tests ask learners to apply domain knowledge to a given problem. Examples of application tests are problem solving tasks such as the application of a theory to a case scenario. For example, learners are asked to apply their knowledge about force and motion to calculate a car’s braking distance. Instruments that consisted of both types of measures and did not report separate results were categorized as mixed tests. For the outcome variable collaboration skills, we distinguished between the assessment of negotiation skills, information sharing skills, coordination skills, and mixed skills based on the collaborative problem solving framework by Liu et al. (2015). We categorized instruments that assessed the quality of skills related to negotiation and argumentation, such as the quality of discussion or argumentation sequences, as negotiation skills. Measures that assessed the quality of information sharing, such as the formulation of individual arguments, were defined as information sharing skills. Measures that assessed the quality of meta-cognitive activities, such as planning and monitoring collaborative strategies, were classified as coordination skills. Finally, measures that assessed components of more than one skill were categorized as mixed skills.

Negotiation scripting: We coded whether the CSCL script prompted or facilitated negotiation among participants. We coded CSCL script activities as negotiation scripting if they were related to argumentation and negotiation, such as resolving conflict, reaching a consensus, compromising, formulating critiques, discussing content-related aspects, and engaging in argumentation sequences. However, prompts about how to construct an individual argument were not classified as negotiation but rather as information sharing prompts.

Information sharing scripting: We assessed whether the CSCL scripts prompted information sharing among collaborators. Information sharing prompts are defined as prompts and hints related to the interchange of information, knowledge, ideas or problem solutions. Examples of scripting information sharing are recognizing differences in the knowledge base of oneself and one’s learning partner, initiating and requesting information sharing processes, and constructing individual arguments without engaging in a discourse.

Coordination scripting: We defined activities prompted by the CSCL script as coordination scripting if they related to the management of group processes or taking on responsibility. For example, learners might be explicitly prompted to coordinate among themselves by discussing learning goals or strategies, or the CSCL scripts replaces such activities by distributing roles or sequencing the learning process into different phases.

Combinations of collaborative activities: Based on the coding for each specific collaborative activity, we further coded a) the number of different collaborative activities prompted by the CSCL script (i.e., one, two, or three different collaborative activities), and b) the specific combinations of collaborative activities or single collaborative activity prompted by the CSCL script. The different types of combinations were negotiation and information sharing, negotiation and coordination, information sharing and coordination, a combination of all three collaborative activities, or only one of the three collaborative activities.

Statistical analyses

Hedges’ g (Hedges 1981) was chosen as a statistical index for all effect sizes reported in this meta-analysis. Furthermore, a random-effects meta-analysis was conducted in which studies were weighted according to their precision defined by the inverse of their variance (Borenstein et al. 2009), as we assume that the true effect size varies between studies, as the treatment (i.e., the CSCL scripts), experimenters, task, sample composition, and other partially unknown covariates all vary between studies. Some effect sizes were statistically dependent because studies reported multiple outcomes for the same sample, or several experimental groups were compared to one control group. We addressed such dependencies by calculating a synthetic effect size for statistically dependent effects (Borenstein et al. 2009). To identify heterogeneity within the data, the ratio of the observed variance between studies to the within study error (Q) was calculated. The ratio of estimated true variance to total observed variance was reported using the I2 statistic. Based on suggestions by Higgins et al. (2003), the amount of heterogeneity is considered low if 25–49% of the observed variance is due to estimated true variance, moderate if 50–74% is estimated true variance, and high if at least 75% of the observed variance is estimated true variance. The moderator analysis was conducted as proposed by Borenstein et al. (2009): A meta-regression with categorical moderator variables was conducted first. This tests the potential moderators by indicating whether Hedges’ gs for each level of a moderator differ significantly from each other. In a second step, Hedges’ g, its confidence interval, and the amount of heterogeneity within the level were calculated for each moderator level.

This meta-analysis was based on published articles. To detect and correct for a possible publication bias, trim’n’fill (Duval and Tweedie 2000) and the three-parameter selection model (3PSM, McShane et al. 2016) were used.

All analyses were conducted in R for Windows 3.6.2 (R Core Team 2019). The metafor (Viechtbauer 2010) package was used to aggregate the comparisons and to calculate the effect sizes. According to Cohen (1988), effect size values ranging between 0.20 and 0.49 are interpreted as small effects, effect size values between 0.50 and 0.79 are interpreted as medium effects, and effect size values of 0.80 or larger are interpreted as large effects.

Results

Summary effect of CSCL scripts on motivation

In total, the included studies reported 33 effects of CSCL scripts on motivation, of which 9 effects were independent. The analyses yielded a non-significant positive effect with a Hedges’ g of 0.13 (see Table 1).

Table 1 Summary Effect Size and Mean Effect Sizes for Moderator Levels for the Effect of CSCL Scripts on Motivation
Full size table

The 3PSM resulted in an adjusted Hedges’ g of 0.14 (SE = 0.10, 95% CI [−0.06; 0.35], p = .17), which is not significantly different from the unadjusted Hedges’ g (X2(1) = 0.03, p = .86). The trim’n’fill method resulted in 2 missing studies on the right side and suggested an adjusted significant Hedges’ g of 0.16 (SE = 0.07, 95% CI [0.01; 0.31], p = .03; see Fig. 1a). These results indicate that CSCL scripts do not negatively affect motivation and that the results might be underestimated due to publication bias.

Fig. 1.
figure 1

a-c. Funnel plots for the effect of CSCL scripts on a) motivation, b) domain learning, and c) collaboration skills.

Full size image

Summary effect of CSCL scripts on domain learning

In total, we examined 124 effects of CSCL scripts on domain learning, of which 54 were independent. Overall, the meta-analysis resulted in a small significant positive summary effect, with Hedges’ g of 0.24. When distinguishing the particular types of domain learning, CSCL scripts showed small significant positive effects on both recall tests (Hedges’ g = 0.27) and application tests (Hedges’ g = 0.21). Studies reporting mixed tests for domain learning yielded a positive but non-significant summary effect of Hedges’ g = 0.09 (see Table 2).

Table 2 Summary Effect Size and Mean Effect Sizes for Moderator Levels for the Effect of CSCL Scripts on Domain Learning
Full size table

Concerning the publication bias analyses for the summary effect of CSCL scripts on domain learning, the 3PSM resulted in an adjusted Hedges’ g of 0.21 (SE = 0.06, 95% CI [0.08; 0.33], p < .01). The Likelihood Ratio Test yielded no significant difference between the adjusted and the unadjusted model (X2(1) = 0.37, p = .54). Trim’n’fill suggested 2 missing studies on the left side of the distribution (see Fig. 1b), with a corrected Hedges’ g of 0.22 (SE = 0.04, 95% CI [0.14; 0.31], p < .01). These results indicate that CSCL scripts positively affect the recall and application of domain learning, although the results could have been slightly influenced by publication bias.

Summary effect of CSCL scripts on collaboration skills

We examined 41 effects of CSCL scripts on collaboration skills. Of these, 23 were independent. The meta-analysis resulted in a significant medium-sized positive summary effect of Hedges’ g = 0.72. The effect sizes for two studies (Stegmann et al. 2012; Weinberger et al. 2010) were noticeable larger than the average effect size. Therefore, all analyses were conducted again excluding these studies. The resulting significant positive summary effect without these studies was also of medium size (Hedges’ g = 0.59). When distinguishing between the different types of collaboration skills measures, the results again revealed significant positive effects of CSCL scripts on each of the three measures. For the effect of CSCL scripts on negotiation skills, the analyses resulted in a medium-sized Hedges’ g of 0.59. For information sharing skills, the analyses resulted in a medium sized Hedges’ g of 0.60 (outliers excluded). For mixed measures of collaboration skills, the analyses resulted in a medium-sized Hedges’ g of 0.52 (see Table 3).

Table 3 Summary Effect Size and Mean Effect Sizes for Moderator Levels for the Effect of CSCL Scripts on Collaboration Skills
Full size table
Table 4 Moderator Levels, Dependent Variables, and Individual Effect Sizes per Study
Full size table
Table 5 Absolut Numbers of Independent Effect Sizes for the Combination of Dependent Variables and Moderator Levels Within One Statistical Comparison
Full size table

The 3PSM for the overall effect of CSCL scripts on collaboration skills (including outliers) resulted in a medium-sized adjusted Hedges’ g of 0.76 (SE = 0.22, 95% CI [0.33; 1.18], p < .01) which is not significantly different from the unadjusted Hedges’ g (X2(1) = 0.05, p = .83). In line with this result, the trim’n’fill suggested no missing studies (see Fig. 1c). The 3PSM for the overall effect of CSCL scripts on collaboration skills (excluding outliers) resulted in a medium-sized adjusted Hedges’ g of 0.62 (SE = 0.18, 95% CI [0.27; 0.97], p < .05), which is not significantly different from the unadjusted Hedges’ g (X2(1) = 0.05, p = .82). Trim’n’fill suggested no missing studies, indicating no publication bias. These results indicate that CSCL scripts positively affect collaboration skills, with no evidence of publication bias.

Moderator effects of number of different collaborative activities scripted

To answer the question of whether scaffolding a combination of different types of collaborative activities influences the effectiveness of CSCL scripts for recall and application tests, we compared CSCL scripts prompting one, two, or three different types of collaborative activities (RQ 3). For this analysis, only the number of different types of collaborative activities was relevant; we did not distinguish between the specific types of collaborative activities scaffolded by the CSCL script.

For domain learning, the effectiveness of CSCL scripts did not significantly depend on the number of different collaboration prompts (Q(5) = 3.19, p = .67). The amount of residual heterogeneity was significant (Q(52) = 114.51, p < .01). Descriptively, the highest effects were found for CSCL scripts prompting solely one type of collaborative activity (either information sharing, negotiation, or coordination), with higher effects for application tests compared to recall tests. The positive effect of CSCL scripts prompting two different types of collaborative activities was only significant for application tests, not for recall tests. CSCL scripts that prompted all three types of collaborative activities had a small significant positive effect on both recall tests and application tests (see Table 2).

The effect of CSCL scripts on collaboration skills did not depend significantly on the number of collaborative activities prompted (Q(2) = 2.87, p = .24). The amount of residual heterogeneity was significant (Q(24) = 108.54, p = < .01). When excluding the outliers, the effect of CSCL scripts on collaboration skills also did not depend significantly on the number of distinct collaborative activities prompted (Q(2) = 0.55, p = .76). The amount of residual heterogeneity was significant (Q(22) = 78.67, p = <.001). Descriptively, the results show that CSCL scripts prompting only one type of collaborative activity were most effective, followed by CSCL scripts prompting two and three types of collaborative activities (see Table 3).

Moderator effects of specific combinations of collaborative activities

To answer the question of whether scripting certain specific collaborative activities and combinations thereof affect the effectiveness of CSCL scripts, we compared CSCL scripts that scripted only negotiation, information sharing, or coordination to CSCL scripts that scripted the respective two-way or three-way combinations (RQ 4). For the effect of CSCL scripts on collaboration skills, the number of studies did not allow for conducting a moderator analysis with such a large number of moderator levels. Therefore, we conducted this analysis only for domain learning outcomes.

For domain learning, the effect of CSCL scripts did not depend significantly on the specific combination of collaborative activities (Q(6) = 5.84, p = .44). The amount of residual heterogeneity was significant (Q(54) = 100.75, p = < .01). Descriptively, the results show that for studies scaffolding only one type of activity, scaffolding coordination is most effective for enhancing domain learning, followed by scaffolding information sharing, while scaffolding only negotiation resulted in a non-significant effect size. When scaffolding a combination of two different collaborative activities, CSCL scripts scaffolding both negotiation and information sharing led to the second highest positive effect size, while the combination of information sharing and coordination is least effective, with a null effect on domain learning. CSCL scripts with a combination of all three types of collaboration skills resulted in the second smallest, yet still significant small effect on domain learning (see Table 2).

Discussion

The present meta-analysis of studies comparing collaborative learning supported by a CSCL script with unstructured collaborative learning did not yield evidence for a strong negative effect of learning with CSCL scripts on motivation. This finding contradicts the repeatedly formulated hypothesis that learning with CSCL scripts might be too coercive, reducing learners’ autonomy and thus leading to a loss of motivation compared to externally less structured collaborative learning (e.g., Dillenbourg 2002; Wise and Schwarz 2017). Moreover, the meta-analysis shows that CSCL scripts have a small positive effect on domain learning and a medium to large positive effect on learning collaboration skills. Here, the inclusion of more recent primary studies confirms the results of a previous meta-analysis on CSCL script studies (Vogel et al. 2017). Within domain learning, CSCL scripts have a stronger effect on recall tests than on knowledge application tests. These findings are in line with theoretical assumptions and empirical findings on learning and transfer in CSCL contexts (Jeong et al. 2019). Within collaboration skills, the differences between negotiation and information sharing skills were rather small. Remarkably, no studies investigated the effect of CSCL scripts on coordination skills.

To analyze the mechanisms that are assumed to be responsible for the effectiveness of CSCL scripts, we compared CSCL scripts that prompted one, two, or three different collaborative activities as well as specific combinations of these activities based on a collaborative problem solving framework (Liu et al. 2015). Most studies prompted either one or all three collaborative activities. Generally, the results show that prompting only one specific activity tends to result in higher effect sizes compared to prompting a combination of two or three collaborative activities. However, this was more strictly the case for collaboration skills than for domain learning. For domain learning, combining two collaborative activities yielded even smaller effects than combining all three collaborative activities. When conducting more differentiated analyses comparing different combinations of particular collaborative activities, we found that particularly the combination of information sharing and coordination led to a small effect size. Among CSCL scripts that prompted only one collaborative activity, coordination scripting yielded the largest effect on domain learning. Among CSCL scripts prompting two collaborative activities, CSCL scripts combining negotiation and information sharing scripting were most effective. For collaboration skills, detailed analyses on the effect of scripting different combinations of collaborative activities were not possible due to the small number of studies. The results show, however, that scripting a greater number of different collaborative activities reduces the effectiveness of CSCL scripts for learning collaboration skills. However, the moderator analyses did not lead to a substantial reduction in heterogeneity between the CSCL script studies. Therefore, it is still necessary to identify other mechanisms that are relevant for the effectiveness of learning with CSCL scripts beyond the mechanisms proposed and analyzed in the present meta-analysis.

Overall effect of learning with CSCL scripts on motivation (RQ 1)

The first research question addressed the effect of collaborative learning with CSCL scripts compared to unstructured collaborative learning on motivation. In the light of the frequent critique that CSCL scripts might be too coercive and thus reduce learners’ autonomy and motivation, it could be hypothesized that learning with CSCL scripts should have a negative effect on motivation (Dillenbourg 2002; Wise and Schwarz 2017). In contrast, the results showed a small positive but non-significant effect of learning with CSCL scripts on motivation with all studies either reporting non-significant or significant positive effects. Taking a closer look at the primary studies reporting effects of CSCL scripts on motivation enabled us to identify some patterns within the data. Interestingly, most studies considering effects on motivation used CSCL scripts that distributed roles among participants. There were, however, striking differences between the studies yielding positive and null effects. Most studies reporting non-significant effects used CSCL scripts that distributed roles among participants such as tutor and tutee or note-taker roles (G.-Y. Lin 2020; Peterson and Roseth 2016; Weinberger et al. 2005). In these cases, the participants were all undergraduate students. However, some CSCL scripts could even positively affect learners’ motivation. For instance, a CSCL script that did not affect motivation in synchronous learning settings, had positive effects on motivation in in asynchronous learning settings (Peterson and Roseth 2016). Other motivating CSCL scripts applied a “natural” role distribution resulting from true knowledge interdependency (e.g. collaboration between psychologists and physicians, Rummel et al. 2009), or distributed roles with different responsibilities among school pupils (Taylor and Baek 2019). In the latter case, the pupils were more motivated when the roles rotated between learning sessions than when fixed roles were used. It seems plausible that undergraduate students have higher prior knowledge regarding collaboration than school students and therefore perceive such artificial roles as more constraining and disruptive of their natural collaboration than fourth and fifth grade students (Fischer et al. 2013). Nevertheless, the CSCL scripts included in this meta-analysis did not have negative effects on motivation, but rather were comparable to unstructured collaboration. Hence, these CSCL scripts were not detrimental, as suggested by critics, but seemed not to exploit their full potential for increasing motivation.

The duration of the intervention could be an alternative factor explaining differences in the effects of CSCL scripts on motivation. It seems plausible that during an intervention with CSCL script collaboration skills develop. Hence, learners could perceive a CSCL script as more coercive after having received support for some time. Therefore, due to the small number of effects we qualitatively compared studies with respect to their intervention duration. The intervention of two studies lasted for several weeks (Peterson and Roseth 2016; Taylor and Baek 2019). Two other studies used interventions that lasted for 60 to 80 min (Rummel et al. 2009; Weinberger et al. 2005). One study did not provide any information on intervention duration (G.-Y. Lin 2020). Although one would expect learners to become less motivated the longer they learn with a CSCL script, the included studies do not support such pattern. Studies using CSCL scripts in long term interventions yielded significant positive effects (Taylor and Baek 2019) on motivation as well as null effects (Peterson and Roseth 2016). Studies that used CSCL scripts in short term interventions resulted in null effects (Rummel et al. 2009; Weinberger et al. 2005). Therefore, the existing data does not allow for conclusions on how motivation is affected by CSCL scripts over time. Other factors in the design of CSCL scripts might affect learners’ motivation such as fading or adapting the CSCL scripts to individual needs. However, to our knowledge no study on CSCL scripts addresses these aspects together with the intervention duration. Thus, it is necessary to address systematically the question of how different CSCL scripts affect motivation, how this effect changes over time, and how technology can help to exploit the full potential of CSCL scripts.

Another explanation for the non-significant effect on motivation might be that learning with CSCL scripts has ambivalent effects on different factors influencing learners’ motivation. CSCL scripts might help learners easily achieve strong feelings of competence and relatedness because they provide a structure for collaborative learning processes and learners’ involvement in a social context. Such feelings of relatedness and competence are connected to higher levels of motivation (Rienties et al. 2012; Ryan and Deci 2000). On the other hand, the coercive nature of CSCL scripts, which strictly define the activities learners are expected to engage in, could lead to a lower degree of autonomy, which in turn reduces motivation (Wise and Schwarz 2017). This combination of positive and negative effects of learning with CSCL scripts on motivation might balance out, leading to a non-significant effect size close to zero. Unfortunately, the small number of studies empirically examining the effect of CSCL scripts on motivation precludes more nuanced quantitative analyses. Thus, more primary studies addressing hypotheses that take a more differentiated view on motivational factors, for example with respect to the basic psychological needs, are needed. Critique of CSCL scripts rarely address the different aspects of motivation. If we regard motivation as a holistic construct, there is no meta-analytical evidence for an overscripting effect. Thus, this criticism remains a postulate without corresponding empirical evidence.

Overall effects of learning with CSCL scripts on domain learning and learning collaboration skills (RQ 2)

Based on the theoretical assumptions of the script theory of guidance, we further hypothesized that learning with CSCL scripts should have a positive effect on domain learning and learning collaboration skills (Fischer et al. 2013). The results support this hypothesis. Thus, we can conclude that CSCL scripts do indeed support the learning of beneficial collaboration processes that eventually lead to better elaboration of the learning content and ultimately to better collaboration skills and domain learning outcomes (King 2007).

As already detected in the previous meta-analysis by Vogel et al. (2017), the effect of learning with CSCL scripts on collaboration skills was substantially higher than the effect on domain learning. In the script theory of guidance, it is assumed that the guidance provided by an external script helps learners to participate in a specific CSCL practice, building and reconfiguring internal scripts they can then recall in other situations (Fischer et al. 2013). In doing so, the CSCL script helps learners engage in collaborative activities that are beneficial for domain learning. Therefore, substantially higher effects on collaboration skills might be due to a more direct link between CSCL scripts and the development of collaboration skills, while domain learning is more indirectly supported by accomplishing what the script suggests learners to do. The wide range in magnitude of the effects of CSCL scripts on collaboration skills, however, raises questions about how effective such scaffolding can be. Although the average effect size of CSCL scripts on collaboration skills is comparable to the effect sizes other scaffolds such as example-based learning have on learning (J. Chen et al. 2018; Jeong et al. 2019; Wittwer and Renkl 2010), some studies report effect sizes far beyond the usual effects of scaffolding (Noroozi et al. 2013c). These studies reveal a need for closer examination and challenge future CSCL script designs to increase the effectiveness for domain learning to a similar size. One reason for the extraordinary effectiveness of these CSCL scripts might be that they do not only provide support during collaborative phases but also ask learners to individually prepare for the joint learning phases. Prior studies have found the combination of individual and collaborative phases during collaborative learning to be more beneficial than individual learning or collaborative learning alone (Olsen et al. 2017). Individual phases allow learners to prepare for collaboration and give them time to think about and prepare their contributions before being engaging in communication with the learning partner, when answers are expected to be formulated immediately and little time for individual thinking is available. Another reason might be that these studies measure the internal collaboration script that was addressed by the CSCL script particularly well, whereas other studies with lower effects use broader measures.

Explaining the effectiveness of CSCL scripts prompting different combinations of collaborative activities (RQ 3 and 4)

For domain learning, we assumed that combining prompts for different types of collaborative activities would increase domain learning through synergistic scaffolding. Prior research has shown that learners who engage in all types of collaborative activities have the highest domain learning outcomes (Andrews-Todd and Forsyth 2018). Hence, successful synergistic scaffolding should lead to effects on learning outcomes when the CSCL script combines prompts for different collaborative activities above and beyond the effects achieved through separate prompts for each collaborative activity (Tabak 2004). To investigate this issue, we compared CSCL scripts that prompted one, two, or three different collaborative activities. In contrast to our expectations, CSCL scripts were descriptively most effective when prompting only one collaborative activity and least effective when prompting a combination of two collaborative activities. Notably, on all levels CSCL scripts were more effective in fostering domain learning as measured by application tests compared to recall tests. This is particularly surprising given that lower overall effect sizes were found for application measures compared to recall measures. This may indicate that exclusively prompting one collaborative activity is most effective for enhancing application-oriented knowledge and skills. One reason for the larger effect might be that performing these collaborative activities evokes higher-order cognitive processes that allow learners to connect the new information with prior knowledge and apply new information to a problem (Chi 2009). Why this is only valid for CSCL scripts that prompted only one or three collaborative activities remains a subject for further research.

The finding that CSCL scripts prompting one collaborative activity outperform CSCL scripts prompting all three types of collaborative activities indicates that CSCL scripts did not successfully induce synergistic scaffolding. One plausible reason for this lack of synergistic scaffolding might be connected to the fact that CSCL scripts that prompt different types of collaborative activities are increasingly demanding. It is possible that these pose an additional load on the learner; in particular, scripts for several collaborative activities pose an even higher cognitive load on learners (e.g., F. Kirschner et al. 2009). Possible solutions to take some load off learners when working with highly complex CSCL scripts might be to offer the scaffolds for the different types of collaborative activities independently of each other or in a specific sequence (Schwaighofer et al. 2017). However, this explanation would be in conflict with the finding that scripting three collaborative activities is more effective than scripting two collaborative activities. Primary research on how to combine prompts for different types of collaborative activities in one CSCL script can lead to synergistic scaffolding (Tabak 2004) is still at a nascent stage, and more research is needed to find the most beneficial design for such scaffolding.

To explore the effectiveness of specific combinations of collaborative activities, we compared the different combinations of collaborative activities prompted by the CSCL script in more detail. CSCL scripts addressing solely coordination or a combination of negotiation and information sharing were most effective, followed by scripting information sharing only and the three-way combination. Notably, the effect of CSCL scripts that only prompted negotiation was very variable and non-significant. Upon closer examination, the variability of this effect is reflected in the variability of the CSCL scripts used in this sample. The CSCL scripts range from very elaborate, highly structured discussion scripts (Noroozi et al. 2013c) to argumentation scripts sequencing the order of arguments and counter-arguments (Stegmann et al. 2007) and CSCL scripts that solely prompt to discuss a specific topic (Rau et al. 2017). Comparing these studies, it seems that CSCL scripts offering a higher degree of structure have larger effects on domain learning. Comparing the different combinations of collaborative activities also provided more detailed insights into the question of how synergistic scaffolding might be achieved (Tabak 2004). Specifically, combining the negotiation and information sharing prompts yielded a higher effect size than offering scaffolding for one of the two types of collaborative activities alone. Analogously to the interpretation of the positive effect of combining of individual and collaborative activities (Olsen et al. 2017), this could be seen as a successful combination of two activities that can lead to synergistic effects. CSCL scripts that only prompted negotiation yielded a non-significant effect, which was dramatically improved by combining negotiation with information sharing. Conversely, scripting information sharing alone already had a significant positive effect, yet the effect was even higher when combined with negotiation. Examining the CSCL scripts used in these studies in detail, it stands out that scripting negotiation only means that students are specifically prompted to engage in discussion; however, an information exchange phase is missing (e.g., Puhl et al. 2015; Wu et al. 2019). This phase might help students better engage in beneficial negotiation activities by encouraging prior listening and thinking about the information their learning partners share with them. This effect seems to be in line with the importance of individual phases in which students can first think about and establish their viewpoint before engaging in collaboration (Olsen et al. 2017). Conversely, the effectiveness of information sharing prompts when learning with CSCL scripts does not seem to be comparably dependent on negotiation prompts, since CSCL scripts scaffolding information sharing only already achieved a substantial effect. It seems that the information exchange is one of the most beneficial activities for learning with CSCL scripts. However, its effectiveness can be diminished by additionally scaffolding coordination but further boosted by including negotiation prompts.

When combining all three types of collaborative activities, any benefits resulting from two-way combinations of scaffolding in CSCL scripts seem to vanish. This could be a consequence of over-loading students with too many different activities to focus on (F. Kirschner et al. 2009). Nevertheless, since each type of collaborative activity investigated seemed to work successfully at least when offered separately or in combination with one other activity, it remains an important avenue for further research to determine how the scaffolding of different collaborative activities should best be combined to increase domain learning. Particularly, it would be interesting to gain more insight into how scaffolding coordination can remain beneficial when combined with other activities. Here, the results of the meta-analysis showed that although CSCL scripts scaffolding coordination alone were quite effective, combining coordination activities with other prompts led to far smaller or even non-significant effects. Overall, the results indicate that it is more important to focus on which specific combination of collaborative activities is prompted by the CSCL script than the number of different collaborative activities prompted by the script. Unfortunately, the low number of studies preclude more detailed analyses of whether the three collaborative activities differ in their potential to advance application-oriented and recall-oriented domain learning. In particular, there is still a lack of primary research on the combination of two different collaborative activities.

For the learning of collaboration skills, the three-way combination also led to the smallest effect size. This means that the CSCL scripts included in this meta-analysis failed to induce differentiated scaffolding of collaboration skills in a beneficial way (Tabak 2004). Ideally, prompting all of the different types of collaborative activities would lead to strong effects of the specific prompts on the corresponding collaboration skills. Thus, the differentiated scaffolding of different collaborative activities within a single CSCL script should not reduce their effectiveness. Consequently, the results of this meta-analysis could be interpreted either as suggesting that differentiated scaffolding is not possible in the way proposed by Tabak (2004), or that it is necessary to further study how to induce differentiated scaffolding when developing CSCL scripts combining scaffolds for different types of collaborative activities in order to support collaboration skills. In addition, the intervention period of most primary studies included in this meta-analysis was rather short. Therefore, it is possible that scripting several different collaborative activities over a short period of time overwhelms the learners by inducing a high cognitive load (e.g., F. Kirschner et al. 2009). It seems plausible that repeated practice of a single collaborative activity falls short if combined with other activity prompts. This might also be a question of measurement. Although CSCL scripts often address more than one collaborative activity, only a few studies measured a mixture of collaboration skills. Therefore, it seems plausible that those scripting only one or two collaborative activities were better able to measure the learning of collaboration skills as their measures better aligned with the specific skill scaffolded by prompting the respective collaborative activity. Also, the prompting of specific types of collaborative activities might not have addressed the optimal scripting level (Fischer et al. 2013). Designing CSCL scripts with a combination of prompts for different types of collaborative activities might often result in prompting only higher scripting levels to avoid scripting that is too extensive and overwhelming scripting. For example, learners might be prompted to discuss the most plausible solution (Rummel et al. 2009) or exchange information (Ertl et al. 2006). Here, particularly inexperienced learners could require more detailed scripting (i.e., on a lower scripting level) for these activities.

Overall, the results of the meta-analysis show that CSCL scripts are beneficial for domain learning and for enhancing collaboration skills. However, the proposed mechanisms, that generally prompting a combination of negotiation, information sharing, or coordination might explain the effectiveness of learning with CSCL scripts, were not supported by the results of this meta-analysis. In particular, combining prompts for two or more types of collaborative activities led to lower effect sizes than only prompting one of these activities. Thus, neither differentiated nor synergistic scaffolding could be successfully achieved by generally combining prompts for different types of collaborative activities in one CSCL script. Thus, for the effectiveness of CSCL scripts, the studies included in this meta-analysis indicate that in some cases less is more when it comes to scaffolding different collaborative activities. This might be due to the additional cognitive load posed by CSCL scripts incorporating scripting for different types of collaborative activities. Thus, how to achieve differentiated and synergistic scaffolding (Tabak 2004) when combining different scaffolds in one CSCL script remains an open question. Sequencing and fading scaffolds throughout the application of a CSCL script might be a promising approach, and thus should be examined in future experimental studies on learning with CSCL scripts.

Implications for the critique that CSCL scripts decrease motivation by “overscripting”

Although motivation is vividly discussed as a factor responsible for the small or negative effects of learning with CSCL scripts (Dillenbourg 2002; Wise and Schwarz 2017), the number of CSCL script studies measuring motivation is rather small. Nevertheless, the non-significant overall effect of CSCL scripts on motivation reported in this meta-analysis does not support the repeatedly asserted critique that the strict structuring of collaborative learning through CSCL scripts leads to a reduction of autonomy, which in turn negatively influences motivation and ultimately impedes learning (Dillenbourg 2002). If such an effect exists at all, it might be reduced by an opposing effect of CSCL scripts enhancing learners’ feelings of competence and social relatedness, which should lead to higher motivation (Järvelä et al. 2010). The empirical studies included in this meta-analysis suggest that CSCL scripts are not detrimental for motivation, but rather can have positive effects. This strongly indicates that the overscripting effect, which is originally based on a conceptual article (Dillenbourg 2002), has been overblown by the research community without being based on empirical evidence. Nevertheless, theories about supporting collaborative learning scenarios could be further developed by reflecting in more detail on the effect of structuring CSCL on motivation and integrating different factors that might have positive or negative effects on motivation, such as those proposed by self-determination theory (Deci and Ryan 1985). Future research on learning with CSCL scripts should measure aspects of motivation by default in order to achieve a more robust sample of effect sizes for motivation. Moreover, given the hypothesized detrimental effect of learning with CSCL scripts on motivation, future CSCL script designs should try to increase the positive effect of CSCL scripts on feelings of competence or social relatedness. Increasing the freedom afforded by CSCL scripts, for example by fading them out (e.g., Wecker and Fischer 2011) or adapting them to learners’ needs (Rau et al. 2017), could presumably decrease a possible negative effect of CSCL scripts on learners’ autonomy.

Implications for the script theory of guidance

The positive effects of CSCL scripts on both domain learning and learning collaboration skills are in line with the theoretical assumptions that learning with CSCL scripts induces beneficial collaborative processes, ultimately leading to better learning compared to unstructured collaborative learning (King 2007). These findings justify the principles formulated in the script theory of guidance that providing external scripts enables learners to engage in collaborative practice in a way that leads to learning of knowledge and skills (Fischer et al. 2013). Although CSCL scripts are mainly considered useful for learning to collaborate (Wise and Schwarz 2017), there are far more studies investigating their effect on domain learning. The small positive effect on domain learning is clearly stable, whereas the large positive effect on collaboration skills is deeply heterogeneous. Additionally, the relatively small number of studies analyzing effects on collaboration skills precludes more comprehensive moderator analyses that could help identify factors explaining the heterogeneity in effect sizes. Thus, in future research, fewer studies on the general effects of CSCL scripts on domain learning are needed, but more studies explicitly analyzing differential effects of CSCL scripts using different designs, CSCL scripts inducing different types of activities, or CSCL scripts implemented in various contexts. In contrast, more studies examining the general effect of CSCL scripts on collaboration skills are still needed. There is a particular lack of studies concerning the effect of CSCL scripts on coordination, despite the fact that studies facilitating coordination were most effective for domain learning. Moreover, given the various measures used to assess collaboration skills, the field would be strengthened by developing instruments for assessing particular collaboration skills such as information sharing or negotiating and widely applying them in studies on computer-supported collaborative learning.

Implications for the design of CSCL scripts aiming at differentiated and synergistic scaffolding

This meta-analysis has shown that combining prompts for different types of collaborative activities led to a reduction in the effectiveness of learning with CSCL scripts, particularly for learning collaboration skills. Consequently, the general functioning of differentiated scaffolding and synergistic scaffolding is doubtful (Kollar et al. 2014; Noroozi et al. 2013c; Tabak 2004). However, more detailed analyses of specific combinations of the three collaborative activities showed that combined scaffolding does not always lead to smaller effects. Thus, some types of combinations might be needed to take full advantage of learning with collaboration scripts. At the very least, a major future goal of scaffolding and scripting research should be to examine how and when CSCL scripts lead to successful differentiated and synergistic scaffolding. Two possible ways to increase the effectiveness of CSCL scripts with combined scaffolds would be to introduce sequencing and adapt the different scaffolds. In an experimental study concerning CSCL scripts for mathematical argumentation, Schwaighofer et al. (2017) showed that presenting different scaffolds in a specific sequence and fading out the scaffold presented first can increase the effectiveness of CSCL scripts, particularly for learners with weaker cognitive prerequisites. In another study, Wecker and Fischer (2011) showed that peer-monitored fading increases the effectiveness of CSCL scripts. It seems plausible to assume that these results are transferrable to scaffolding different collaborative activities within one CSCL script.

Limitations

Of course, this meta-analysis is not without limitations that must be considered when interpreting the results and drawing conclusions and implications. Although there is a large body of research and strong theoretical foundation concerning the effectiveness of CSCL scripts on learning, the design and context of CSCL scripts in empirical research varies greatly. The targeted collaboration skills range from the construction of individual arguments in short one-hour trainings to engaging in argumentative discourse, exchanging peer feedback and preparing mutual individual introductions, and even adhering to assigned roles for several weeks of collaborative learning activities. This leads to difficulties in finding comparable CSCL script studies and summarizing them in a reasonable way. In addition, there is wide variety in what constitutes a CSCL script. However, by applying a consistent definition of CSCL scripts and searching for studies on CSCL scripts without solely using the term script, but rather focusing on the mechanisms of CSCL scripts, we tried to identify an appropriate sample of primary studies representing what has been discovered empirically about learning with CSCL scripts. It was, however, necessary to constrain the literature search to specific search terms. Although we also used other terms than the term “script” itself, it is possible that we systematically missed studies from research areas using a different nomenclature. One such area might be the research on dynamic support such as conversational agents. We included two studies using conversational agents for the present meta-analysis (Adamson et al. 2014; Ulicsak 2004). Our decision to not use the rather broad search term “support” has led to missing out studies from this field of research (e.g., Wang et al. 2011). However, the respective analyses did not indicate any substantial publication bias. This leads us to conclude that we found a comprehensive sample of primary studies for the phenomenon under investigation. Nevertheless, it is possible that specific research areas are underrepresented.

Regarding the coding of the primary studies, a major problem is that most studies do not provide direct indicators for the factors that are theoretically and empirically assumed to positively affect the effectiveness of learning with CSCL scripts. The assumption is that the more learners engage in specific activities in the learning process (e.g., negotiation), the more they should benefit from collaborative learning (Chi and Wylie 2014; Liu et al. 2015). To analyze the effectiveness of learning with CSCL scripts based on this assumption, the most direct indicator primary studies could provide would be measures of the activities learners actually engage in throughout the collaborative learning process. However, most studies do not report such measures; thus, the most proximal information provided by the primary studies is the description of activities prompted by the CSCL script. What actually happened during the collaboration process in the different studies can only be estimated using this information, resulting in several sources of uncertainty. First, the accuracy of this information varies across studies. While some studies report the activities required in the CSCL script in detail (e.g., Rummel et al. 2009; Ulicsak 2004), others are less detailed, which might have led to inaccurate estimations of the learning activities that were actually used in the studies’ learning processes (e.g., Hsu et al. 2015; Tsovaltzi et al. 2015). A second source of uncertainty is that, even when the activities required by the CSCL scripts are described in detail, we do not know to what extent the learners adhered to what they were asked to do during the learning process. Sometimes learners only complete the required activities on a very superficial level. Nevertheless, the description of activities required by the CSCL scripts is the most proximal estimation for the collaboration process available. Moreover, we assume that the presumed inaccuracy of this measure is well distributed across studies and thus should not substantially bias the results of the comparisons between different types of CSCL scripts.

Another limitation of this meta-analysis is that, although motivation features prominently in criticisms of learning with CSCL scripts, there were only a few studies measuring motivation (e.g., Peterson and Roseth 2016). Moreover, motivation is conceptualized in CSCL research in highly diverse ways, such as approaches building on expectancy value conceptualizations, distinguishing between extrinsic and intrinsic motivation or conceptualizations such as self-determination theory integrating the needs for competence, autonomy, and social relatedness (Deci and Ryan 1985). The broad variety of conceptualizations and the rather small number of independent effect sizes weaken our ability to interpret them coherently and do not allow for examinations of further moderating effects.

More generally, the number of CSCL script studies included in this meta-analysis is relatively low, which reduces the possibility of comparing between different studies on a more fine-grained level. The number of studies at each level of the examined factors was sometimes too low or too unevenly distributed to conduct a comparison and interpret the results in a reasonable manner. This was particularly true for studies analyzing the effects of CSCL scripts on collaboration skills and on motivation. Additionally, even within a moderator level the reported effect sizes are often very heterogeneous. This indicates that studies within moderator levels vary systematically due to unknown covariates. Therefore, such results should be interpreted carefully. Unfortunately, we cannot solve this issue until more primary studies are conducted that would allow more fine-grained analyses.

The final limitation that must be considered when interpreting the results of the meta-analysis at hand is that comparisons of different levels of the moderators are only comparisons between studies. Almost none of the studies compared the moderator levels as a within-study effect (e.g., Peterson and Roseth 2016). Thus, the differences between different moderator levels cannot be interpreted causally. They only suggest a direction for the empirical relationship, which might have been confounded by the specific study designs.

Conclusion

This meta-analysis investigated the effect of CSCL scripts on motivation, domain learning, and collaboration skills and proposed moderators to explain their effectiveness. Most importantly, this meta-analysis shows that the widely proposed hypothesis (Dillenbourg 2002; Wise and Schwarz 2017) that CSCL scripts reduce learners’ motivation is not supported by the available data. Although a negative effect of CSCL scripts on motivation seems very unlikely given the present data, the existing empirical evidence might still not be enough to refute this criticism completely. Only a few studies empirically investigated the effect on motivation at all, and more differentiated analyses comparing different CSCL scripts’ effects on motivation are lacking. It seems plausible that CSCL scripts foster some aspects of motivation, such as a feeling of competence, and at the same time diminish other aspects of motivation, such as fulfillment of the need for autonomy. Therefore, we call for a systematic empirical investigation of how different CSCL scripts affect the various aspects of motivation and how a possible influence interacts with the learning setting (e.g., duration of the intervention, synchronous versus asynchronous learning, learners’ age, learners’ prior knowledge). We suggest self-determination theory (Deci and Ryan 1985) as a potential theoretical foundation for obtaining a differentiated picture on motivational aspects that are important for students’ learning.

Furthermore, the results are in line with prior studies’ findings showing that CSCL scripts indeed have positive effects on domain learning and learning of collaboration skills. Hence, CSCL scripts engage learners in beneficial activities that enhance learning (Fischer et al. 2013). However, the results of this meta-analysis suggest that future research should focus on developing detailed measures of collaboration skills instead of simply replicating the well-established effect of CSCL scripts on domain learning. Although the effect of CSCL scripts on collaboration skills is larger compared to their effect on domain learning, it is also particularly heterogeneous, indicating that the effect differs widely between different types of CSCL scripts. Hence, future research may focus on identifying moderators that explain this heterogeneity. A further promising area of research would be to investigate how effects of CSCL scripts can be increased by designing dynamic CSCL scripts.

One aspect that seems to influence the effectiveness of CSCL scripts at least descriptively is the number of different collaborative activities prompted. For collaboration skills, this meta-analysis revealed that when it comes to designing CSCL scripts, less is more. Scaffolding fewer different types of collaborative activities generally led to higher effects of CSCL scripts on collaboration skills. For domain learning, these effects were less clear. The hypothesized effectiveness of combining different scaffolds and thus leading to differentiated and synergistic scaffolding (Tabak 2004) did not pan out for some combinations of collaborative activities. Hence, future research is needed that investigates how and which collaborative activities could be combined to enhance the effectiveness of CSCL scripts. Additionally, sequencing and fading different prompts within a single CSCL script might better exploit the potential of combining scaffolds that have been shown to be beneficial when being applied alone. However, the present meta-analyses failed to identify clear moderators explaining this variance.

Overall, the positive effect of CSCL scripts on domain learning and collaboration skills highlight the potential of CSCL scripts to facilitate learning. The finding that motivation is not significantly affected by learning with CSCL scripts stands in contrast to the wide-spread belief that CSCL scripts undermine learners’ agency. Altogether, CSCL scripts seem to enable learners to make use of the different benefits of collaborative learning rather than hinder their learning processes.