Introduction

Students collaborating in computer-based learning environments often have problems regulating their learning (Azevedo and Hadwin 2005; Manlove et al. 2006). They often do not engage in enough metacognitive activities to control and monitor their learning. Metacognitive scaffolding can support students' metacognitive activities and learning (Azevedo et al. 2008; Land and Green 2000; Veenman et al. 2005). However previous scaffolding studies only examined the effects of scaffolding on students' learning (Molenaar et al. 2010; Veenman 2011). Therefore, there is little in-depth knowledge of how metacognitive activities are related to types of scaffolding (structuring vs. problematizing) and learning. Unlike past studies on a data set that examined whether the post-intervention outcomes or the group’s metacognitive activities simply differ across the control and experimental conditions (Molenaar et al. in press), this study uses multivariate, multilevel methods on a subset of the data to test an explanatory model of the relationships among the scaffolds, student activities (metacognitive and others), group mate activities, and individual learning.

Moreover, most research into scaffolding focuses on the effects of metacognitive scaffolds in individual settings (Azevedo et al. 2008; Veenman et al 2005). Although these results can be used to understand the role of metacognitive scaffolding on student learning in collaborative learning settings, some important issues related to the nature of collaborative learning need further exploration. In small groups, students elaborate, discuss and give feedback on one another’s contributions, which supports learning (Chi 2009; Van Boxtel 2004; Van Drie and Van Boxtel 2004; Webb 2009). Furthermore, student involvement is important, a student’s constructive activities affect learning more than active activities attending to other student’s contributions (Chi 2009). Consequently, to understand the effect of scaffolding in collaborative settings, it is crucial to understand how scaffolds influence student involvement embedded in the group’s interaction. In addition, the underlying assumption of constructivist theories is that the nature of learning activities (e.g. cognitive, metacognitive activities) influences student learning (Duffy and Jonassen 1992; Janssen et al. 2010). During collaborative learning, many activities beyond metacognitive activities (such as cognitive, relational and off-task activities) support students’ learning (Janssen et al. 2010). Therefore, we will argue that a comprehensive analysis of how metacognitive scaffolding affects learning requires that the other learning activities are taken into account to assess the unique effects of metacognitive activities.

This paper examines the question: to what extent do metacognitive activities mediate the effects of different scaffolds on students’ learning? We argue that different forms of metacognitive scaffolds foster metacognitive activities differently and, in turn, will have differential effects on student learning, i.e., students’ metacognitive knowledge and domain knowledge. We expect that metacognitive activities mediate this relationship between scaffolding and individual learning. The activities of 54 students during their 51,339 conversation turns across 108 hours analyzed as they collaborated face-to-face in triads in a computer supported learning environment. There were three metacognitive scaffolding conditions (none, structuring, and problematizing). We used mixed methods, namely discourse analysis and multi-level statistical analysis. As such, this paper not only contributes to our understanding of how different metacognitive scaffolds affect students’ metacognitive activities and learning, it also offers practical insights on how to create scaffolds that support students’ engagement in activities that aid learning.

The effect of metacognitive scaffolding on metacognitive activities and learning

Scaffolding is defined as providing assistance to a student when needed and fading the assistance as the competence of the student increases (Wood et al. 1976). Research indicates that scaffolding facilitates learning as it supports learners in tasks that they cannot accomplish successfully by themselves, as well as developing knowledge for future learning (Hmelo-Silver and Azevedo 2006; Pea 2004; Sharma and Hannafin 2007). Metacognition is defined as knowledge about and regulation of cognitive activities (Flavell 1979). Metacognitive scaffolding aims to help students to adequately control and monitor their learning (Azevedo et al. 2008; Molenaar et al. 2010; Veenman et al. 2005). Students in small groups are supported to engage in metacognitive activities, such as orientation, planning, monitoring, evaluation and reflection (Meijer et al. 2006). Research showed that metacognitive scaffolds in small groups stimulates metacognitive activities and enhances students’ learning (Azevedo and Cromley 2004; Land and Green 2000). Researchers often assume that metacognitive activities mediated the effect of scaffolding on learning, but there is little empirical evidence for this assumption (Veenman et al. 2005). Moreover, scaffolding and metacognitive activities are often embedded in interaction among the group members. To understand how metacognitive scaffolding affects students' learning during collaboration, we look at perspectives on collaborative learning.

Collaboration can aid student learning when students modify their knowledge through interactions within their group. Various collaborative learning perspectives e.g., cognitive elaboration, Chi 2009; Mercer 1996; Webb 2009; van Boxtel 2004; socio-cognitive conflict, Piaget 1932; Doise 1990; Doise and Mugny 1984; co-construction, Hatano 1993; van Boxtel 2004) stress different mechanisms that cause learning during collaboration (giving, receiving and using explanations; resolving conflicts; co-construction). They all emphasize that students’ elaborations on one another’s contributions support learning. Thus, a side effect of metacognitive scaffolding in small groups is that the interaction among the group members can stimulate reflection, provide feedback and elicit discussion of metacognitive activities, which in turn enhances individual learning (Chi 2009; Webb 2009).

Another important issue influencing learning in collaborative settings is a student’s involvement in the learning activities. Active vs. constructive vs. interactive learning activities are each related to different cognitive processes (Chi 2009). Active activities entail attending to ongoing actions through activating prior knowledge, assimilating new knowledge and storing it (Chi 2009). Stronger involvement is found in constructive activities, in which a student goes beyond the presented information through self-explaining, inferring new knowledge, and organizing or restructuring existing knowledge (Mayer & Wittrock, 1996; Chi 2009). Finally, in interactive activities, students build on their group members’ contributions through elaboration, feedback, agreeing and challenging ideas (Chi 2009; Webb 2009). For example, studies have shown that even collaboration with an ignorant partner generates better learning achievements than learning alone (Chi 2009). Ignorant partners pose questions that elicited their partner’s constructive activity. Furthermore, in pairs of students with similar past achievement, in which each student performs as an “explainer” or a “listener” role, the explainers learn more than the listeners (Coleman et al. 1997; Hausmann et al. 2004; Schwartz and Bransford 1998). By engaging in more constructive activities than the listeners, the explainers benefit more from their participation in collaborative activity. This indicates that even though interaction among group members supports learning during collaboration, the student’s involvement in these activities influences how he or she learns. Thus, students who engage in more constructive activities due to scaffolding might benefit more than students who engage only in active activities.

Based on the above research, it can be argued that scaffolding in a collaborative setting may foster student involvement embedded in interaction among the group members, which in turn, affects students’ learning. Reiser (2004) specified two mechanisms to explain how scaffolding affects student learning. Structuring simplifies the learning assignment by reducing its complexity, clarifying the underlying components and supporting performance (i.e., providing the students with an example of a plan for the assignment). Problematizing increases the complexity of the learning assignment by emphasizing certain aspects of the assignment and asking learners to clarify the underlying components and perform actions to construct their own strategies (i.e., asking students to make their own plan for the assignment). These different mechanisms support the formation of different scaffolds that either structure or problematize metacognitive aspects of the learning assignment.

Structuring scaffolds give context suitable examples of metacognitive activities to the group (e.g., showing students an exemplary plan for their mind mapping task when they start this task “What would you like to learn; let’s make a mind map with important topics to learn, for instance the climate”). Structuring scaffolds encourage students’ attention to the information in the scaffold, but do not invite them to construct their own metacognitive activities. On the other hand, problematizing scaffolds pose context suitable questions that elicit students’ metacognitive activities (e.g., asking students to plan their mind mapping task when they start this task “How are you going to make the mind map?”). Past studies showed that problematizing scaffolds such as question prompts elicit students’ explanations and support articulation of students’ thinking (Chi et al. 2001; Davis and Linn 2000; King 1998, 2002). Thus, problematizing scaffolds are likely to encourage students’ constructive activities.

Different scaffolds could influence student involvement differently. Scaffolds that drive the students’ interaction could stimulate metacognitive activities beyond the direct impact of the scaffolding. Interaction among the group members can further stimulate metacognitive activities when students start to elaborate, discuss and reflect on one another’s contributions. Referring back to the example of the structuring planning scaffold, students can elaborate on this example, adjusting and shaping the group’s plan for the mind map task. In response to the problematizing scaffolds, students can have discussions about (conflicting) views, exchange, share, or co-construct metacognitive activities together.

To conclude, metacognitive scaffolding can influence student learning through supporting and stimulating metacognitive activities that monitor and control the group’s cognitive activities. Different scaffolds provide different supports for metacognitive activities, possibly stimulating student involvement embedded in the interaction between the group members differently. Unlike scaffolding in an individual setting, student interaction scaffolding in a collaborative setting modifies student involvement and supports additional metacognitive activities which can influence learning. The next section elaborates on the effect of metacognitive activities on learning in collaborative settings.

Effects of metacognitive activities on learning during collaboration

In the section above, we argued that metacognitive scaffolding can stimulate metacognitive activities, which in turn aids student learning of domain and metacognitive knowledge (Veenman 2005; 2011). Metacognitive activities monitor and control cognitive activities, which directly address the task content; for example, students read, elaborate and process information in discussions. Students who engage in more cognitive activities acquire more domain knowledge (Chinn et al. 2000; Howe et al 2007). Metacognitive activities support the development of domain knowledge through activating prior knowledge, planning the use of effective strategies to obtain learning goals, integrating new knowledge with existing knowledge, monitoring the group’s activities in relation to the learning goals, and evaluating understanding. Such metacognitive activities optimize the cognitive activities, which aids student learning of domain knowledge.

Metacognitive activities support student’s metacognitive knowledge through showing examples, providing room for practice and receiving feedback (Veenman 2011). Group members construct metacognitive activities in reciprocal interaction (Iiskala et al. 2004; Iiskala et al. 2010). Moreover, metacognitive activities embedded in intensive interaction among the group members support productive metacognitive decisions (Goos et al. 2002). These interactions are likely to also help develop students’ metacognitive knowledge (Salomon 1993; Veenman 2011; Molenaar et al. in press). Students in groups can share existing metacognitive knowledge and build on one another’s metacognitive contributions to co-construct new metacognitive knowledge (Lin and Sullivan 2008; Iiskala et al. 2010). Their metacognitive activities can elicit new activities from the other group members. These activities offer opportunities for further metacognitive activities and allow students to appropriate knowledge from other group members. Subsequently, these activities can aid students’ developing knowledge and alter their future participation, which in turn can contribute to the knowledge development of other group members (Salomon 1993; Volet et al. 2009). As noted above, student involvement varies across activities (Chi 2009). Students’ own activities are often constructive; whereas attending to other group members’ contributions often only requires their attention (Chi 2009). Thus we argue that a student’s own metacognitive activities are more likely than attention to other group members’ metacognitive activities to influence his or her metacognitive knowledge.

Apart from cognitive and metacognitive activities in the problem content space, students in small groups engage in activities in the social relational space (i.e., motivating one another, engaging one another and managing allocation of tasks; Janssen et al. 2010; McGrath 1991). The group’s activities in the relational space can enhance group members’ social relationships, aid their collaboration and facilitate their learning. These relational activities foster a positive group climate, increase group cohesion, and aid task completion (Kreijns et al. 2003; Massey et al 2003; McGrath 1991; Jehn and Shah 1997; Wilson et al. 2006). Likewise, negative socio-emotional processes such as rudeness, insults or domination reduce the quality of group solutions (Chiu and Khoo 2003; Webb et al. 2002). Off-task activities (e.g., discussing weekend plans) in the social relational space can improve relationships among group members, but they also tend to reduce learning and achievement (Chiu 2004). Accordingly, cognitive activities and metacognitive activities support the development of knowledge, while relational activities foster a positive group climate that can support learning. In contrast, off-task behaviors often hinder learning. Hence, multiple activities must be modeled when analyzing the effects of scaffolding and metacognitive activities on learning.

The present study

The purpose of this study is to examine the relationships among different scaffolds, metacognitive activities and students’ learning in a collaborative learning setting. To our knowledge, there are few empirical studies available on the effects of scaffolding on learning in a group setting that also accounts for both the learning activities and the learning outcomes. We report an experiment with three metacognitive scaffolding conditions (none, structuring, and problematizing). The main question addressed in this study is: To what extent do metacognitive activities mediate the effects of metacognitive scaffolding and different scaffolds (structuring vs. problematizing) on students’ domain and metacognitive knowledge? This question entails three hypotheses:

  1. Hypothesis 1.

    Scaffolding and different scaffolds support student’ domain and metacognitive knowledge

    Previous studies have shown that scaffolding improves student learning. Therefore, we expect that students supported by scaffolding will outperform students in the control group on both domain and metacognitive knowledge. As problematizing scaffolds are more likely than structuring scaffolds to foster constructive metacognitive activities, we expect students supported with problematizing scaffolds to outperform those supported with structuring scaffolds on both domain and metacognitive knowledge.

  2. Hypothesis 2

    Scaffolding and different scaffolds support metacognitive activities

    Previous studies have shown that scaffolding stimulates metacognitive activities. Thus, we expect more metacognitive activities from students receiving scaffolding than those who do not. As problematizing scaffolds explicitly elicit students’ metacognitive activities and stimulate interaction among students, we expect more metacognitive activities from students who receive problematizing scaffolds than those who receive structuring scaffolds.

  3. Hypothesis 3.

    Metacognitive activities support student’ domain and metacognitive knowledge

    Finally, we argued that metacognitive activities support students’ domain knowledge and metacognitive knowledge. As outlined above, student involvement in learning activities influences their learning. A student’s own activities are likely to aid learning more as they are often more constructive than simply attending to other group members’ contributions. Therefore, we expect that a student’s own metacognitive activities are more important than group members’ metacognitive activities in mediating the relationship between metacognitive scaffolding and individual learning.

Figure 1 shows a path diagram of the hypothesized relationships.

Fig. 1
figure 1

Overview of the relations studied

Full size image

Methods

Subjects

For this study, we analyzed the learning activities of students. Due to the labor-intensive nature of discourse analysis, we could not analyze all triads that participated in the full study (Molenaar et al. 2010). In the full study, 156 students in three schools divided over six classes participated. The teachers assigned students to triads (52) to maximize heterogeneity. Teachers rated students as low, middle or high achievers based on their reading, writing and computer abilities and then created triads containing one low, one middle and one high achiever, with at least one boy and one girl. We randomly assigned the triads to the three experimental conditions: (a) no scaffolds (control group, 16 triads); (b) structuring scaffolds (experimental group 1, 17 triads); and (c) problematizing scaffolds (experimental group 2, 19 triads). The conditions were equally divided over the classes. By using randomly assigned triads to the conditions within a class, we blocked for effects of classes.

As coding all conversation turns from all triads requires enormous time, labor and resources, we randomly drew a smaller sample of 18 triads (one in each scaffolding condition from each class) for this study. The sample consists of 54 students (23 boys and 31 girls) assigned in six control triads, six triads in the structuring condition and six triads in the problematizing condition. The students of this sample were in Grade four (9), Grade five (27) or Grade six (18) across six classes in three elementary schools. These three schools were comparable, all in outer city suburban areas with a white middle class population. Within each class, equal numbers of triads were assigned to the different conditions, resulting in an equal allocation of triads in each scaffolding condition across schools. For a sample size of 54, an effect size of 0.4 and a significance level of p = .05, the statistical power is 0.86. Hence, non-significant results at the individual level must be interpreted cautiously.

Procedure

Virtual learning environment and assignment

The e-learning environment in this study, Ontdeknet, supports students in their virtual collaboration with experts (Molenaar 2003). The experts shared information about their country with students that were edited by the editor of Ontdeknet. The teacher gave the assignment and monitored students’ progress. Collaborative learning is implemented at two levels: students collaborating with each other face-to-face in small groups with a computer and with an expert in a virtual environment. The study consisted of eight lessons, each lasting one hour. In the first lesson, the students completed a pre-test, and then received instructions about the assignment and the virtual environment. In the last lesson, the students completed several post-tests. All students received the same instructions, and all triads spent the same time working on the assignment (six hours). During these six lessons, the triads worked on an assignment called “Would you like to live abroad?” The goal of the assignment was to explore a country of choice (New Zealand or Iceland), write a paper on their findings, and decide if they would like to live in that country. The triads worked on one computer and had access to an expert, namely an inhabitant of the country. They could consult the expert by asking questions and requesting information about different topics in the country. In a separate expert window in the computer environment, the expert provided the requested information, and questions were answered in a forum. Four sub-tasks preceded the task of writing a paper about the country: (a) introducing the group to the expert, (b) writing a goal statement, (c) selecting a country, and (d) specifying topics of interest on a mind map. All tasks were integrated into the working space of the triads, where they also wrote the paper. The performance of each triad was stored in the learning environment. All lessons were supervised by the same researcher.

The scaffolding system and the conditions

Scaffolds are messages that support the learner in tasks that they cannot successfully perform without help (Wood et al. 1976). Both forms of metacognitive scaffolds were dynamically integrated into the computer environment. The triads of students in both experimental scaffolding conditions received computerized scaffolds supporting their metacognitive activities during the first two lessons at the same instance in the learning process (Molenaar and Roda 2008). These scaffolds were given when metacognitive activities are typically executed in the learning process. The timing was based on Zimmerman’s model for self-regulated learning (Zimmerman 2002). The computerized scaffolding system determined the appropriate instance to send a scaffold based on the students’ attention focus. Students in the scaffolding conditions received a minimum of 12 scaffolds in each condition. The triads in the structuring condition (experimental group one) received direct support for their metacognitive activities; for example, the computer avatar David showed the students an exemplary plan of a task “The expert would like to know what you want to learn. Please write all the topics about New Zealand that you would like to learn more about in this mind map” (see Fig. 2). In response, students can elaborate and reformulate the specifications to the planning activities of the group, see Fig. 2. The triads in the problematizing condition (experimental group two) received scaffolds designed to elicit students’ metacognitive activities and explanations; for example, the computer avatar David asks, “How are you going to make a mind map?” The triads in the problematizing condition were obliged to answer the avatar’s questions in an answer box on the screen, see Fig. 2. In response, students can construct a plan of how to make a mind map. Lastly, the control group triads saw the avatar David, but did not receive any metacognitive scaffolds (to control for a Hawthorne effect, in which the avatar’s mere presence could influence the student activities, Franke and Kaul 1978).

Fig. 2
figure 2

Example of structuring (left) and problematizing (right) scaffolds

Full size image

Measurements

The learning activities

The conversations within each triad of students were audiotaped with voice-recorders. We coded the transcribed protocols of each lesson. The unit of analysis was the conversation turn of each speaker. Each conversation turn was coded with one main category code, see Table 1 for an overview and one subcategory code, see Appendix A Tables 5, 6 and 7. All main categories were mutually exclusive and exhaustive categories, as were all subcategories within a main category.

Table 1 Main categories of our coding scheme
Full size table

Several categories (cognitive activities, metacognitive activities, off task activities, not codable activities and teacher activities) were derived from the coding scheme of Veldhuis-Diermanse (2002). Additionally, two types of activities were added; relational activities specific for the group setting and procedural activities specific for our learning environment. In this analysis, we focus on cognitive, metacognitive, relational and off-task activities. The cognitive activity category contains turns regarding the content of the task and elaboration of this content (e.g., reading the material, asking a question about the domain, discussing the learning task, elaborating specific issues and summarizing previous contributions of group members, see Appendix A Table 5). Metacognitive activity includes turns that monitor or control cognitive activities, and includes Meijer et al. (2006) subcategories: orientation, planning, monitoring, evaluation and reflection (see Appendix A Table 6). Relational activity includes turns regarding the social interaction among students, such as engaging other group members, discussing the division of labor among the group members, and supporting other group members (see Appendix A Table 7). Off task refers to activities that are not related to either the learning task at hand or the task domain, and teacher activities are contributions made by the teacher.

To determine the inter-coder reliability, two raters independently coded two randomly selected protocols (2500 turns). There was an excellent agreement for the main categories (Fleiss 1981): Cohen’s kappa = 0.92. The kappa was highest for the metacognition category, k = 0.94, and lowest for the non-codable category, k = 0.82. The dataset consists of 51,339 activities at the conversation turn level across 108 hours of discourse.

Using these codes, we computed individual’s and group mates’ proportions of turns; for example, % My cognitive activities = person’s cognitive turns / total turns of group; % Group mates’ cognitive activities = group mates’ cognition turns / total turns of group

We computed parallel pairs of variables for each main category. Furthermore, we analyzed all the responses of the triads to the scaffolds to select representative excerpts illustrating how the triads generally responded to the scaffolds.

Individual learning achievements

The individual learning achievements were assessed by measuring each student’s domain and metacognitive knowledge on separate tests. Domain knowledge was measured by a curriculum-based knowledge test with 40 questions (true/false/question mark) about the country the students had studied. Students received one point for each correct answer, and zero points for a question mark or an incorrect answer. The question mark option was included to reduce guessing, as we told the students that for each incorrect answer, one point would be subtracted from their test score. Cronbach’s alpha was 0.93 for the New Zealand test and 0.88 for the Iceland test. This test was also used as a pre-test before students engaged in the learning assignment. The time between pre-test and post-test was eight weeks.

The metacognitive knowledge of the students was measured by asking them to imagine that they were going to do the same assignment again. They were asked to write down the steps that they would take to do this assignment. The answers were scored against a full procedural overview made by the researchers. The full procedural overview consisted of 18 steps; examples of steps were “plan the learning task”, “activate prior knowledge” and “monitor the activity of the group.” The maximum score was 18 points. Ten percent of the tests were scored by two independent researchers (kappa = 0.83). We did not conduct a pre-test for metacognitive knowledge as the test was not suitable for that purpose.

Analysis

We used mixed methods to analyze the conversations of students in the different conditions. To understand how different scaffolds stimulate metacognitive activities among students, we used discourse analysis (Gee 2005). We selected representative excerpts of conversations in which students responded to different forms of metacognitive scaffolds, illustrating how they stimulate students’ metacognitive activities, how students respond (active, constructive or interactive activities) and how they influence students’ interactions.

To test these hypotheses, we must address analytical difficulties involving these outcome variables and these explanatory variables (see Table 2). There are two outcome variables, and they differ across groups and across individuals. To analyze the two outcome variables simultaneously (domain and metacognitive knowledge), we used a multivariate outcome model to account for contemporaneous correlation in the errors across equations (Goldstein 1995). To model differences across groups and across individuals simultaneously, we used a multilevel analysis (aka Hierarchical Linear Modeling, Bryk and Raudenbush 1992; Goldstein 1995) to account for heteroskedasticity.

Table 2 Addressing each analytical difficulty with a statistics strategy
Full size table

The explanatory variables may show indirect, mediation effects or false positives. To test for multilevel, mediation effects, we used a multilevel, mediation test (Krull and MacKinnon 2001). Testing many hypotheses increases the likelihood that at least one of them incorrectly rejects a null hypothesis (false positive). To control for this false discovery rate, we used the two-stage linear step-up procedure, which outperformed 13 other methods in computer simulations (Benjamini et al. 2006).

The hypotheses were tested through a three-step process. First, we studied the influence of the control variables on test scores. Second, we conducted regression analyses to test the direct effect of different forms of metacognitive scaffolds on test scores and metacognitive activities. Finally, we tested the mediating influence of metacognitive activities on the relationship between metacognitive scaffolding and test scores with multi-level mediation tests (Krull and MacKinnon 2001).

We estimated a multivariate, multi-level regression of the following form:

$$ {\mathbf{Tes}}{{\mathbf{t}}_{{igy}}} = {\beta_{{00y}}} + {{\mathbf{f}}_{{0gy}}} + {e_{{igy}}} $$
(1)

β00y are the grand mean intercepts of Test igy , a vector of y outcome variables (domain knowledge test score and metacognitive knowledge test score) for student i in group g. The group- and student-level residuals are f 0gy and e igy respectively.

This study design seeks to control for students’ abilities and gender influences. Specifically, each triad includes a student with high ability, one with medium ability, and one with a low ability. Furthermore, each triad included at least one girl. Regressions confirmed that neither domain knowledge test score nor metacognitive knowledge test score were associated with means or distributions of ability or gender.

To examine the link between scaffolding interventions and test scores, we entered a vector of u scaffolding conditions: structuring and problematizing (Scaffold) with the control group as the baseline. Each set of predictors was tested for significance with a nested hypothesis test (χ2 log likelihood, Kennedy 2004).

$$ {\mathbf{Tes}}{{\mathbf{t}}_{{igy}}} = {\beta_{{00y}}} + {{\mathbf{e}}_{{igy}}} + {{\mathbf{f}}_{{0gy}}} + {\beta_{{sjy}}}{\mathbf{Scaffol}}{{\mathbf{d}}_{{igy}}} + {\beta_{{tgy}}}{\mathbf{Tur}}{{\mathbf{n}}_{{igy}}} $$
(2)

Then, we entered a vector of x variables indicating specific conversation turn characteristics: total group turns, percentage of conversation turns in which a student engaged in each activity in their triad (total individual turns, cognitive activities, relational activities, procedural activities, and off-task activities, Turn), and percentages of the above activities of other group members.

Next, we tested whether the metacognitive scaffolding conditions were linked to the percentage of conversation turns in which a student engaged in metacognitive activities in a triad.

$$ {\text{Metacognitio}}{{\text{n}}_{{igy}}} = {\beta_{{00y}}} + {{\mathbf{e}}_{{igy}}} + {{\mathbf{f}}_{{0gy}}} + {\beta_{{sgy}}}{\mathbf{Scaffol}}{{\mathbf{d}}_{{igy}}} + {\beta_{{tjy}}}{\mathbf{Tur}}{{\mathbf{n}}_{{igy}}} $$
(3)

Lastly, we added the percentage of conversation turns in which a student engaged in metacognitive activities in a triad (Metacognition) to Eq. 2. By doing this we can test our third hypothesis.

$$ {\mathbf{Tes}}{{\mathbf{t}}_{{igy}}} = {\beta_{{00y}}} + {{\mathbf{e}}_{{igy}}} + {{\mathbf{f}}_{{0gy}}} + {\beta_{{sgy}}}{\mathbf{Scaffol}}{{\mathbf{d}}_{{igy}}} + {\beta_{{tgy}}}{\mathbf{Tur}}{{\mathbf{n}}_{{igy}}} + {\beta_{{1gy}}}{\text{Metacognitio}}{{\text{n}}_{{igy}}} $$
(4)

We used multi-level mediation tests across the above vectors (Krull and MacKinnon 2001). For significant mediators, the proportional change was 1 – (b'/b), where b’ and b were the regression coefficients of the explanatory variable, with and without the mediator in the model, respectively.

We reported how a 10% increase in each continuous variable above its mean was linked to each outcome variable. As percent increase is not linearly related to standard deviation, scaling is not warranted.

An alpha level of .05 was used. Testing many hypotheses increases the likelihood that at least one of them incorrectly rejects a null hypothesis (false positive). To control for the false discovery rate, we used the two-stage linear step-up procedure, which outperformed 13 other methods in computer simulations (Benjamini et al. 2006).

Results

We start with our findings from the discourse analysis. By discussing two representative excerpts, we show how different forms of metacognitive scaffolds influence students’ responses. Next, we illustrate how metacognitive activities influence peer interactions and foster domain and metacognitive knowledge. Lastly, we report the findings of the multivariate, multi-level analyses.

Discourse analysis of representative excerpts

First, we look at the student responses to different scaffolds. A structuring scaffold is typically followed by either the implementation of the scaffold example or a group discussion elaborating on the example. On the other hand, problematizing scaffolds elicit student activities, leading the group to elaborate, share knowledge, resolve a conflict or co-construct new metacognitive activities. We illustrate this with two examples, the structuring scaffold excerpt in example one and the problematizing excerpt in example two.

Speaker

Code

Conversation turn

Avatar

Structuring scaffold

A learning goal is what you want to learn. For instance, we would like to learn more about New Zealand to decide if we would like to live there.

Paul

Metacognitive

Ok, so we will say

Simon

Metacognitive

We are going to make a paper about

Loes

Metacognitive

We are going to make a paper about Iceland.

Example 1. An example of a response to a structuring scaffold (underlined texts is spoken by the avatar)

After the structuring scaffold, Paul accepted the example given (“Ok”) and started to apply the example to their assignment with a planning activity (“we will say.”). Simon and Loes finished his effort by applying the example of the avatar to their assignment “We are going to make a paper about Iceland.” Unlike the structuring scaffold, the problematizing scaffold in example two stimulated a rich discussion about a learning goal’s meaning, its purpose, and its role, rationale and implementation in this assignment.

Speaker

Code

Conversation turn

Avatar

Problematizing scaffold

How are you going to write down a learning goal?

Mien

Metacognitive

What is a learning goal?

Jan

Metacognitive

A learning goal is what you want to learn, for example I become a president.

Joost

Metacognitive

For example, right?

Jan

Metacognitive

Yes for example there are many things you have to learn.

Joost

Metacognitive

Because we want to know things.

Mien

Metacognitive

Yes, but what do we want to learn now?

Jan

Metacognitive

We want to learn about a country to see if we want to live there.

Joost

Metacognitive

Right, and this we have to explain to the expert.

Example 2. An example of a response to a problematizing scaffold (underlined texts is spoken by the avatar)

The problematizing scaffold ignites a chain reaction of metacognitive activities. First, Mien asks for the meaning of a learning goal (“What is a learning goal?”). Jan answers by defining a learning goal as “what you want to learn” and by giving an example “become a president.” After clarifying that “become president” was an example, Joost claims that the purpose of learning is “because we want to know things.” In response, Mien asks for their immediate learning goal (“but what do we want to learn now”). Jan answers with their learning goal for this assignment (“We want to learn about a county,”) and its rationale (“to see if we want to live there.”) Joost concurs (“Right”) and articulates its implementation, (“this we have to explain to the expert.”). Through their exploration of the learning goal, the group members orient to the task and construct a better understanding of it. Each student’s metacognitive activity triggers another group member’s metacognitive activity. Furthermore, each metacognitive activity provides validating feedback to the previous one and provides grist from which to co-construct the next one, thereby valuing the importance of metacognitive activities and encouraging its subsequent use and development. Thus these two examples illustrate how the metacognitive activities elicited by problematizing scaffolds result in more student involvement and interaction, which seem qualitatively different than the activities stimulated by structuring scaffolds.

Metacognitive activities as mediating mechanism

Next, we illustrate how metacognitive activities mediate student learning during collaboration. First, we show an example that illustrates how metacognitive feedback is given during collaborative learning. Second, we show how metacognitive activities improve student’s cognitive activities.

Metacognitive knowledge is developed through practices, examples and feedback. In example three, Joep contributes a plan (“let’s write down hobbies”) to write the introduction assignment. Eline and Noor immediately start implementing this plan (“My hobbies are tennis and ballet”). Eline’s and Noor’s contributions give feedback to Joep that his planning remark was useful. This feedback may positively influence Joep’s metacognitive knowledge. In example two, there was a more elaborate interaction around metacognitive activities, in which group members actively construct metacognitive activities, but also built on one another’s contributions. In these types of interactions, the elaboration, feedback and co-constructive contributions can help build students’ metacognitive knowledge.

Student

Code

Conversation turns

Joep

Metacognitive

Let’s write down hobbies

Eline

Cognitive

My hobbies are tennis and ballet

Noor

Cognitive

I play the guitar

Example 3. An example of metacognitive activity that is implemented through cognitive activity

With respect to domain knowledge, cognitive activities build and elaborate on the topic studied. Metacognitive activities during interactions monitor and control these cognitive activities as shown in example 4.

Student

Code

Conversation turns

Ine

Cognitive

These are all products of New Zealand… not imported …

Mark

Cognitive

Does have to import.

Ine

Cognitive

New Zealand that does import.

Sophie

Metacognitive

That is wrong, does not import.

Ine

Cognitive

All products of New Zealand, thus not imported.

Mark

Metacognitive

That is a good sentence.

Sophie

Metacognitive

Yes and now it is right, lets continue …

Example 4. An example of cognitive and metacognitive activities

While writing their paper, Ine expressed a new idea (“not imported”) and Mark wrote it down. When Mark misunderstands it (“does have to import”), Ine does not notice and repeats (“does import”). However, Sophie detects and corrects the error (“that is wrong, does not import”). Sophie’s monitoring controls her group mates’ cognitive activities. Next, Ine continues to clarify and correct the sentence (“not imported”). Mark evaluates and accepts it (“good sentence”), and Sophie confirms and plans to continue the formulation of the next sentence. Sophie’s metacognitive activity improves her group’s cognitive activities and receives validation from other group members, which highlights its importance and encourages its further use and development. This instance is likely to help the group members remember that New Zealand does not import all these products, thus affecting the group members’ domain knowledge.

Descriptive findings

Starting with a low domain knowledge pre-test mean of 7.07, the students scored much higher on its post-test (M = 20.72; maximum = 36). Scores on the subsequent metacognitive knowledge test were modest (M = 5.30; maximum = 12). During their group interactions, a student’s activities included many cognitive activities (9% of the triad’s turns on average), metacognitive activities (7%), relational activities (7%) and fewer off-task activities (4%). Other group members engaged in substantial relational activities (14%). (The percentages do not sum to 100% due to codes for group mates’ activities and for other activities, such as procedural activities. See summary statistics in Appendix B, Table 8.)

Multilevel analyses

The variance components multi-level model (intercept-only) for domain knowledge scores showed that 45% of the differences were between groups (suggesting substantial similarity among members of the same group), and 55% were among students within each group (see Table 3). For metacognitive knowledge test scores, 65% of the differences were between groups (also showing substantial similarity among group mates) and 35% were across students within each group.

Table 3 Unstandardized regression coefficients (with Standard Errors) of multivariate, multilevel regression model results simultaneously predicting post-test and metacognitive knowledge test (N = 54)
Full size table

Relation between metacognitive scaffolds and learning

Hypothesis 1 concerned the direct effect of scaffolding and different scaffolds on domain and metacognitive knowledge. Findings indicate that students in the structuring and problematizing condition outscored students in the control condition on the domain knowledge post-test by 2.65 (not significant) and 4.55 (significant) points respectively on average (see Table 3, Domain knowledge, Model one). Furthermore, students whose proportion of cognitive activities exceeded its mean by 10% averaged 6.35 points higher on the post-test (see Table 3, Domain knowledge, Model two). When other group members’ proportion of relational activities exceeded its mean by 10%, a student averaged 6.67 points higher (see Table 3, Domain knowledge, Model two). Controlling for other learning activities (cognitive and relational activities), regression coefficients of the different scaffolds on domain knowledge are only slightly smaller (see Table 3, Domain knowledge, Model three). Students in the problematizing condition still outperformed students in the other conditions.

Students in the structuring and problematizing conditions outscored students in the control condition by 1.98 or 2.19 points respectively on the metacognitive knowledge test on average (see Table 7 in Appendix B, Metacognitive knowledge, model one). Furthermore, students whose proportion of cognition turns exceeded its mean by 10% averaged 2.32 points higher, respectively, on the metacognitive knowledge test (see Table 3, Metacognitive knowledge, model two). In contrast, students whose proportion of off-task behaviors exceeded its mean by 10% averaged 4.31 points lower on the metacognitive knowledge test (see Table 3, Metacognitive knowledge, model two). Controlling for other learning activities (cognitive activities and relational activities), the findings show that the effect of problematizing scaffolds on metacognitive knowledge, although a little bit smaller, is still stronger compared to structuring scaffolds (see Table 3, Metacognitive knowledge, Model three). Controlling for other learning activities, students in the problematizing scaffolds condition still outperformed students in the other conditions.

Hypothesis 2 concerned the effect of scaffolding and different scaffolds on metacognitive activities. The results in Table 4 show that the students receiving metacognitive scaffolding displayed proportionately more metacognitive activities than other students. Students receiving problematizing scaffolds showed slightly more metacognitive activities than students receiving structuring scaffolds, but this difference was not significant.

Table 4 Unstandardized regression coefficients (with Standard Errors) of multivariate, multilevel regression model results predicting the % of metacognition (N = 54)
Full size table

Hypothesis 3 concerned the extent to which metacognitive activities mediate the relationship between different scaffolds and students’ domain knowledge and metacognitive knowledge. The findings show that students whose proportion of metacognition exceeded their mean by 10% averaged 9.06 points higher on the post-test (see Table 3, Domain knowledge, Model four). Controlling for individual proportion of metacognitive actions reduced the problematizing scaffold condition regression coefficient by 35% (multi-level mediation test z = 2.02; p < .05; r = .50; Table 3, Domain knowledge, Models three and four). Together, these explanatory variables accounted for 25% of the domain knowledge post-test score variance.

With regard to the mediating effect of metacognitive activities on metacognitive knowledge, the findings show that students whose proportion of metacognition exceeded their mean by 10% averaged 3.01 points higher on the post-test (see Table 3, Metacognitive knowledge, Model four). Controlling for individual proportion of metacognitive actions reduced the structuring scaffold and problematizing scaffold conditions’ regression coefficients by 86% and 77% respectively (multilevel mediation tests: z = 2.02; p < .05; r = .39; and z = 2.01; p < .05; r = .50). Together, these explanatory variables accounted for 39% of the metacognitive knowledge test score variance.

Discussion

In this study, we examined to what extent metacognitive activities mediated the effect of different scaffolds on students’ domain knowledge and metacognitive knowledge. Three hypotheses were assessed to answer this question. The first hypothesis addressed whether different forms of metacognitive scaffolding affected students’ metacognitive knowledge and domain knowledge. In the structuring condition, the avatar showed contextually suitable examples of metacognitive activities, whereas in the problematizing condition it posed questions to elicit metacognitive activities. Both metacognitive scaffolds (structuring and problematizing) were associated with higher scores on the metacognitive knowledge test. Only problematizing scaffolds were linked to greater domain knowledge; structuring scaffolds did not significantly affect domain knowledge. With regard to the second hypothesis, the findings show that scaffolding stimulated metacognitive activities (the two scaffolds did not differ significantly). Regarding the third hypothesis, students receiving either metacognitive scaffold engaged in more metacognitive activities, which were linked to their higher metacognitive knowledge test scores. Meanwhile, only problematizing scaffolds were linked to greater domain knowledge, and individual metacognitive activities also mediated this relationship.

These findings suggest that both forms of scaffolding affect students’ metacognitive knowledge, mainly through the metacognitive activities that they stimulate. Contrary to our expectations, we did not find a significant difference in the number of metacognitive activities in each scaffolding condition. However, only problematizing scaffolds were linked to greater domain knowledge, suggesting that the metacognitive activities elicited by problematizing scaffolds differed from those elicited by structuring scaffolds. The discourse analysis suggests that structuring scaffolds encouraged students to discuss the application of the example while problematizing scaffolds stimulated students to construct metacognitive activities in interaction with their group members. Hence, structuring scaffolds might foster active metacognitive activities from the students, whereas problematizing scaffolds might trigger more constructive activities embedded in intensive interaction. Constructive activities are likely more effective than active activities at aiding knowledge acquisition (Chi 2009). Thus, this qualitative difference in the student interactions might help explain why problematizing scaffolds were associated with greater domain knowledge, while structuring scaffolds were not. This is an important finding because it suggests that the effect of metacognitive scaffolds on learning operates through both a greater number of metacognitive activities within the group and the student’s own involvement in the metacognitive activities.

Finally, we controlled for other learning activities that can affect learning during collaboration (Janssen et al 2010; McGrath 1991). The analysis showed that both forms of metacognitive scaffolds were associated with greater metacognitive activities without significantly influencing other activities. Yet, other learning activities did influence students’ metacognitive knowledge and domain knowledge. Students performing proportionately more cognitive or metacognitive activities scored higher on both the domain knowledge test and the metacognitive knowledge test, consistent with earlier findings (Janssen et al 2010). However, other group members’ cognitive and metacognitive activities did not significantly contribute to a student’s domain or metacognitive knowledge in this study, in contrast to earlier studies claiming that students’ elaborations on one another’s contributions fosters learning (Chi 2009; Mercer 1996; Piaget 1932; Webb 2009; Weinberger & Fischer, 2006; van Boxtel 2004). This issue needs more attention in future research especially since we know so little about how metacognitive activities embedded in interaction influence students’ learning (Dillenbourg et al 2009; Iiskala et al. 2010).

We did find some evidence that group mates influence a student’s learning. When a student’s group mates performed proportionately more relational activities, the student scored higher on the domain knowledge test. Relational activities were previously found to foster a positive group climate (Kreijns et al. 2003; Massey et al 2003; McGrath 1991; Jehn and Shah 1997; Wilson et al. 2006), but were not yet explicitly connected to learning. This study suggests that group mates’ relational activities (but not one’s own) foster a student’s learning of domain knowledge. An example of how the relational activities can influence the domain knowledge is given in Appendix C. Finally, students who were often off-task scored lower on the metacognitive knowledge test, but not on the domain knowledge test, unlike previous studies linking off task activity with less domain knowledge (e.g., Chiu 2004). Overall, these results highlight the effects of different activities on learning and the importance of distinguishing between the student’s activities and those of group mates.

In summary, the problematizing scaffold is more strongly linked to student learning than the structuring scaffold is, perhaps due to the qualitative differences in their respective students’ metacognitive activities. However, we have not systematically investigated the effects of different scaffolds on student involvement in the group interactions throughout the whole learning assignment. Interaction patterns are often established early in the learning assignment and remain rather stable through the collaboration (Kapur et al. 2008). This could entail that groups supported with problematizing scaffolds continue to show more intensive interaction through the learning assignment. Future research can examine how scaffolding influences the interaction among the group members during earlier and later time periods of their collaboration. Finally as discussed above, metacognitive activities have received relatively little attention in collaborative learning research as an explanatory factor for learning (Dillenbourg et al. 2009). We showed that they influence student domain and metacognitive knowledge in collaborative settings, but further research can examine how they are embedded in interaction and how that influences their monitoring and control of cognitive activities.

Conclusions

In this study, we examined if metacognitive activities mediate the learning effects of metacognitive scaffolding. Our analysis of the discourses and achievements of 54 elementary school students showed that students receiving either form of metacognitive scaffolds (structuring or problematizing) engaged in more metacognitive activities and showed more metacognitive knowledge than students who did not receive any scaffolding. However, only students receiving problematizing scaffolds showed greater domain knowledge, which was also mediated by their own metacognitive knowledge. The discourse analysis suggests that qualitative differences in students’ metacognitive activities can account for the differences between problematizing and structuring scaffolds. These results suggest the superiority of problematizing scaffolds over structuring scaffolds for some tasks.

This study has several limitations regarding sample size, the interaction context of the metacognitive activities, and time/sequence. As this study only has 54 students, non-significant results at the student level must be interpreted cautiously (even though there are 51,339 conversation turns). Meanwhile, we did not examine the micro-time context of recent conversation turns in which metacognitive activities are embedded. One approach to modeling the micro-time context is to examine the characteristics of sequences of recent conversation turns. The impact of a student activity (e.g., metacognition) might differ across micro-time contexts. Likewise, the impact of a student activity (e.g., planning) at the beginning of the session (earlier time period) might differ from the same activity at the end of a session (later time period, Reimann 2009). Lastly, different sequences of the same set of activities (M,C,M,C,M,C vs. M,M,M,C,C,C) may have different effects on student learning (Chiu 2008).

On a practical level, the results suggest that problematizing scaffolds and some activities in collaborative settings can aid learning. Specifically, designing problematizing metacognitive scaffolds into virtual learning environments for some tasks can enhance individual group members’ metacognitive activities to aid acquisition of domain knowledge and metacognitive knowledge more than learning environments with structuring scaffolds or with no scaffolds. Furthermore, instructional designs might enhance individual group member’s domain knowledge by engaging all group members in cognitive and metacognitive activities and encouraging group members to engage in relational activities. Additionally, instructional designs might enhance individual group members’ metacognitive knowledge by engaging all group members in cognitive and metacognitive activities and by reducing their off task behavior.