Abstract
This meta-analysis examined research on the effects of self-regulated learning scaffolds on academic performance in computer-based learning environments from 2004 to 2015. A total of 29 articles met inclusion criteria and were included in the final analysis with a total sample size of 2648 students. Moderator analyses were performed using a random effects model that focused on the three main areas of scaffold characteristics (including the mechanism, functions, delivery forms, mode, and number of scaffolds; how to promote self-regulated learning by scaffolds); demographics of the selected studies (including sample groups, sample size, learning domain, research settings, and types of computer-based learning environments); and research methodological features (including research methods, types of research design, types of organization for treatment, and duration of treatment). Findings revealed that self-regulated learning scaffolds in computer-based learning environments generally produced a significantly positive effect on academic performance (ES = 0.438). It is also suggested that both domain-general and domain-specific scaffolds can support the entire process of self-regulated learning since they demonstrated substantial effects on academic performance. Different impacts of various studies and their methodological features are presented and discussed.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Self-regulated learning (SRL) is regarded as one of the most important competences for lifelong learning (Ifenthaler 2012). It is also the key component for mediating success in most learning environments (Hartley and Bendixen 2001; Lehmann et al. 2014; Zimmerman 2008). The number of studies focusing on SRL has increased rapidly in the past decades. At the same time, the research focus has shifted from the concept of SRL to the applications of self-regulation (van Ewijk 2011). In recent years, more and more researchers have expanded their interests to the promotion of SRL through a range of methods (Dignath and Büttner 2008). These methods include, for example, regulating learners’ motivation and emotion (Efklides 2011) or directly teaching learning strategy (Kistner et al. 2010) in different learning domains. In addition, many researchers have developed scaffolds to facilitate SRL in computer-based learning environments (Azevedo et al. 2004; Dabbagh and Kitsantas 2005; Delen et al. 2014). However, there is still a lack of evidence concerning the effectiveness of SRL scaffolds (Azevedo et al. 2004; Devolder et al. 2012). The current study aims to investigate the effectiveness of computer-mediated SRL scaffolds through an in-depth meta-analysis.
Evidence from the literature shows that high-achieving students have distinct self-regulatory abilities, such as accurate monitoring of learning and adaptation of metacognition (Isaacson and Fujita 2012). There is also agreement on the effectiveness of SRL on academic performance based on numerous empirical studies (Chung 2000; Chen and Huang 2014; Paris and Paris 2001; Zimmerman and Bandura 1994). However, little is known about which aspects of SRL intervention have resulted in the positive impact on academic performance. In this study, we focus on the effects of different kinds of SRL scaffolds on academic performance in computer-based learning environments. The meta-analysis approach was adopted to investigate the effectiveness of SRL scaffolds on academic performance in computer-based learning environments to gain an overview of the evidence on the effects of SRL scaffolds.
Meta-analysis is the quantitative approach to systematic review, and it has been widely used in different fields (Bangert-Drowns and Rudner 1991). An earlier study by Kulik and Kulik (1989) demonstrated that meta-analysis was very useful in gathering information about the effectiveness of interventions. Furthermore, meta-analysis is more systematic than other kinds of reviews (Borenstein et al. 2011), which can help to determine the most effective interventions from a range of options. Importantly, meta-analysis can transform the outcomes of different studies into a common metric, which allows to compare the results of different studies using different instruments and interventions (van Ewijk 2011). In addition, Hedges and Olkin (1986) reported that the overall effect of different studies obtained through meta-analysis was more accurate than the individual studies. However, a meta-analysis of computer-mediated SRL scaffolds remains lacking. This meta-analysis aims to review empirical studies on computer-mediated SRL scaffolds from 2004 to 2015. The following section provides an overview of SRL scaffolds in computer-based learning environments.
Literature review
Self-regulated learning in computer-based learning environments
SRL is defined as an active and constructive process through which learners can set goals, and monitor and control their cognition, motivation, and behavior (Printrich 2000). It is also characterized as a self-directive process as self-beliefs enable learners to transform their academic abilities (Zimmerman 2008). Winne and Hadwin (1998) proposed that SRL included four phases: defining the task, setting goals and plans, enacting tactics, and adapting metacognition. Therefore, learners need to analyze the learning context and define tasks, set the appropriate learning goals and make plans, select the effective learning strategies to use, monitor the whole learning process, and evaluate their learning performance. Based on the evaluation, they may need to modify and adapt certain aspects of their motivation, emotion, cognition, and metacognition (Azevedo et al. 2009).
Previous studies indicated that SRL is the crucial skill for success in computer-based learning environments (Adeyinka and Mutula 2010). However, learners cannot always regulate themselves successfully because of reasons such as lack of good strategy use (Pressley and Hilden 2006), lack of metacognitive knowledge (Veenman et al. 2006), failure to control of metacognitive processes (Hacker et al. 2009), or lack of experience in learning environments with multiple representations (Azevedo et al. 2009). Thus, how to foster SRL ability has become a central issue in the field of education research and practice.
Azevedo et al. (2004) proposed that computer-based learning environments present the promising potential to foster students’ abilities to regulate their learning. Computer-based learning environments provide important opportunities for self-directed learning and allow for a high degree of learner control (Lajoie and Azevedo 2006; Winters et al. 2008). Computer-based learning environments are typically characterized by multiple representations of information that can be controlled by the learner and can help to define tasks, set goals, enact strategies, and adapt metacognition (Winters et al. 2008). Researchers have explored and developed various kinds of computer-based learning environments to scaffold SRL. For example, Perry and Winne (2006) developed gStudy software to capture students’ self-regulated engagements using time-stamped trace data. Azevedo et al. (2010) designed a hypermedia learning environment, Meta Tutor, to detect, trace, and foster SRL about human body systems through pedagogical agents. Narciss et al. (2007) used non-embedded indirect intervention tools for monitoring and evaluation of learning tasks to promote SRL. In addition, assessment of self-regulation can be supported through computer-based learning environments. Aleven et al. (2010) proposed a method for context-sensitive assessment of SRL strategy with an intelligent tutoring system. SRL scaffolds in computer-based learning environments have become increasingly important because students cannot master higher-order thinking without such guidance (Simons and Klein 2007).
Self-regulated learning scaffolds
It is widely acknowledged that the use of scaffolds can support SRL processes (Dabbagh and Kitsantas 2005; Shapiro 2008). Scaffolds can be defined as the process through which learning efforts are supported in open learning environments (Hannafin et al. 1999). Scaffolds can be tools, strategies, prompts, metacognitive feedback, or guides that help students to achieve higher levels of understanding beyond their current abilities (Jackson et al. 1994; Simons and Klein 2007). The theoretical underpinning of scaffolding is that it can provide guidance for students on an as-needed basis and fade as abilities increase (Hogan and Pressley 1997).
Scaffolds can be categorized based on different functionalities. Hannafin et al. (1999) classified four kinds of scaffolds, namely conceptual, metacognitive, procedural, and strategic. Conceptual scaffolds aim to provide guidance about what to consider when a problem is already defined (Hannafin et al. 1999). Metacognitive scaffolds include non-human or human agents who can help students self-regulate the underlying problem-solving process (White et al. 2000). Procedural scaffolds can provide guidance about how to use resources or perform tasks (Azevedo et al. 2001). Strategic scaffolds provide different techniques or methods as well as solution paths for students (Lajoie et al. 2001). Scaffolds can differ in their delivery mode and may be embedded or non-embedded, fixed or adaptive, hard or soft, direct or indirect (Devolder et al. 2012). Embedded scaffolds are integrated in the learning environment so that students are obliged to pay attention to them, while non-embedded scaffolds are initiated by students themselves (Narciss, Proske, and Koerndle 2007). Fixed scaffolds are static and will not adapt to individual students’ needs, while adaptive scaffolds can adjust to meet students’ needs (Azevedo et al. 2004). Hard scaffolds are static or fixed, and soft scaffolds are customizable and negotiable (Saye and Brush 2002). Direct scaffolds are explicit, while indirect scaffolds are tacit (Stahl and Bromme 2009).
To date, however, there is no agreement on the effectiveness of SRL scaffolds. For example, studies revealed that prompts were effective in terms of regulation of student cognition (Lee et al. 2010; Devolder et al. 2012). Moos and Azevedo (2008) found guiding questions that can help students to be aware of and monitor their learning were effective scaffolds.
Concept mapping tools can also serve as effective scaffolds in promoting SRL (Marée et al. 2013). Therefore, researchers designed and developed different kinds of tools using advanced technologies to scaffold SRL. For example, Poitras and Lajoie (2014) developed an agent-based adaptive system for scaffolding SRL in history education by tracing techniques. Lee et al. (2010) found that a combination of generic learning strategy prompts with metacognitive feedback can promote learners’ recall and comprehension in a computer-based learning environment. Lehmann et al. (2014) examined the effectiveness of preflective as well as reflective prompts and indicated that directed preflective prompts worked best for novice learners in an online learning environment.
In summary, there is consensus on the need for scaffolding to promote SRL in computer-based learning environments (Devolder et al. 2012). However, there is still a lack of clarity about the most effective type of scaffold in terms of supporting SRL processes. The present study provides an overview of the effectiveness of SRL scaffolds in computer-based learning environments through the meta-analysis approach.
Findings from earlier meta-analyses of SRL
Meta-analysis can compare the results of different studies and provide the overall effect through one common metric (van Ewijk 2011). To date, meta-analysis of SRL remains limited, and only a few meta-analyses have examined the effectiveness of intervention in the area of SRL. An early meta-analysis by Hattie et al. (1996) aimed to identify features of study skills interventions in 51 studies. Their findings revealed an average effect size of 0.57 for academic performance, 0.16 for study skills, and 0.48 for affect. Later, Benz (2008) investigated the relationship between SRL and academic performance and found the overall effect of interventions on academic performance was 0.78.
Dignath et al. (2008) meta-analyzed 30 studies on fostering SRL at the primary school level and found that SRL training programs are effective. Dignath and Buettner (2008) investigated the impact of different training characteristics on the outcomes by the meta-analysis of 49 studies conducted among primary school pupils and 35 studies with secondary school students. Their results indicated an overall mean effect of 0.69 with no significant difference between the two samples in terms of the mean effects. Sitzmann and Ely (2011) examined how adults regulated their learning of work-related knowledge and skills, and found that goal level, persistence, effort, and self-efficacy were the self-regulation constructs with the strongest effects on learning.
These earlier meta-analyses contributed to insight into the effectiveness of SRL intervention. However, they also highlighted some methodological problems such as studies with no control group, and where initial equivalence between experimental and control groups had not been established (Sitzmann and Ely 2011). In addition, the previous meta-analyses did not clarify the effectiveness of SRL scaffolds on academic performance in computer-based learning environments. Moreover, it was noted that few studies provided empirical evidence about the type of scaffolds that affect academic performance. Therefore, the present meta-analysis was performed to identify and examine SRL scaffolds and their effects, demographics of the selected studies and methodological features. We aimed to overcome the methodological problems and to investigate the following research questions:
-
1.
Which kinds of SRL scaffolds (including mechanism, functions, delivery forms, mode, numbers, and manner in which SRL is facilitated by scaffolds) are the most effective in improving academic performance in computer-based learning environments?
-
2.
What study characteristics (including sample groups, sample size, learning domain, research settings, and types of computer-based learning environments) moderate the effects of SRL scaffolds on academic performance?
-
3.
What methodological features (including research methods, types of research design, types of organization for treatment, and duration of treatment) moderate the effects of SRL scaffolds on academic performance?
Methodology
Meta-analysis procedure
The present study follows the seven steps of meta-analysis proposed by Cooper (2010): (1) define research questions; (2) formulate the selection criteria to include and exclude studies; (3) search the literature; (4) code studies based on the scheme; (5) analyze the effects of interventions; (6) analyze the moderating effects of study characteristics; (7) draw final conclusions and results of the meta-analysis.
Paper selection processes
This study selected papers relevant to SRL scaffolds in computer-based learning environments in the Web of Science database from 2004 to 2015. Previous studies have suggested the Web of Science database as the most popular measure of the quality of the scientific publications quality (Casal et al. 2011). As the study aimed to analyze the evolution of computer-mediated SRL scaffolds in the past decade, the time frame 2004–2015 was selected. The paper selection process proceeded in two steps. In the first step, 450 papers closely related to SRL in the field of educational research were selected using paper title and keywords search (self-regulated learning, SRL, self-regulated learning scaffold) within the Web of Science database. For the second step, the authors scanned the full text of each paper to decide which ones were suitable for inclusion based on the following criteria:
-
1.
Only research articles were included in this study. Book reviews, book chapters, proceeding papers, meeting abstracts, editorials, news, and reviews were excluded.
-
2.
The scaffolds should be computer-mediated and focus on SRL processes and academic performance. Human or peer scaffolds were excluded.
-
3.
Only experimental or quasi-experimental studies related to the effects of scaffolds on academic performance were included. Conceptual studies and studies not reporting academic performance were excluded.
-
4.
The studies need to provide data to calculate effect size (ES), including means and standard deviation, F or t values.
-
5.
Studies could have taken place in any country but needed to be reported in English.
-
6.
Studies should include control groups and provided pretest data.
-
7.
Studies could include students in K–12 and in higher education.
Of 450 papers, only 88 articles met the first three criteria. After rigorous selection by scanning the full text, only 29 articles met the remaining criteria. Greenhalgh and Peacock (2005) indicated that snowball sampling was another useful search strategy. After searching the reference lists of the 29 articles, it was found that no further papers met the criteria. The present study thus included 29 journal papers indexed in the Social Sciences Citation Index of the Web of Science, which were analyzed further. The main reason for this small number was that computer-mediated scaffolds were not always available, or measures of academic performance or the statistical information required for calculating effect size were not always reported.
Coding scheme
Three main study characteristics were examined in detail, including characteristics of the scaffolds, demographics of the selected studies, and methodological features. The scaffold characteristics included the scaffolds’ mechanism, function, delivery form, mode, numbers of scaffolds, and how to facilitate SRL by scaffolds. Demographics of the studies were analyzed across the research sample groups, sample size, research learning domain, research settings, and types of computer-based learning environments (CBEs). Methodological features included research methods, types of research design, types of instruction, and duration of treatment. Table 1 illustrates the coding schemes of each sub-dimension in detail. Some of these sub-dimensions, namely research sample groups, research settings, research learning domains, research methods, types of research design, and types of instruction for treatment, have also been applied in other reviews (Kinshuk et al. 2013; Wang et al. 2014) or meta-analyses (Chang and Lin 2013).
Inter-rater reliability
Two raters independently coded all the studies based on the coding schemes. The agreement rate calculated by Cohen’s kappa was 0.88, which was regarded as reliable (Cohen 1960). The two raters discussed and resolved all discrepancies.
Analysis
To calculate the effect sizes, we adopted the software Comprehensive Meta-Analysis (CMA), developed by Biostat (www.meta-analysis.com). In the present study, moderator analysis was conducted to examine the direction of the differences between subgroups and between the average effect sizes of the variables (Karadağ et al. 2015). The effect size d was calculated by the standardized mean difference, which was equal to the difference between the mean values of experimental and control groups divided by the standard deviation (Hedges and Olkin 1986). Since the outcome variable was academic performance, we only calculated the effect sizes of academic performance. If one study adopted multiple tests to examine academic performance, these effect sizes were averaged to obtain one representative effect size for each study by CMA software. Since a random effects model allows sampling error at the study level (Dignath et al. 2008), a random effects model was adopted to report the effect size, standard errors, confidence intervals, and z and p values in the present study. In addition, the publication bias and sensitivity analyses were conducted by CMA to examine the analysis results.
Results
Overall effect
Hattie’s (2009) definition of effect size was adopted, with an effect size of 0.2 considered as small, 0.4 as medium, and 0.6 as large. Table 2 shows the number of articles (k), effect size (ES), standard error (SE), variance, confidence intervals, z value, p value, and test of heterogeneity in effect size for all articles. The overall weighted effect size was 0.438 for 29 articles with a total sample size of 2648 subjects. The meta-analysis results revealed that SRL scaffolds had a medium but significantly positive effect on academic performance. This effectiveness has been confirmed by the overall weighted effect sizes of the 29 articles. Furthermore, generalization of these results is regarded as very stable and reliable because 2648 subjects were involved in the meta-analysis. The result of the heterogeneity test indicated that the distribution of effect sizes among the articles was significantly heterogeneous (Q = 408.224, df = 28, p = 0.000). This implies that some factors other than sampling error accounted for the variance. Table 3 shows the effect size (ES), standard error (SE), variance, confidence intervals, and z and p values for each study. In the next section, the characteristics of scaffolds, demographics of the studies, and methodological features were examined to investigate the moderating effect.
Characteristics of self-regulated learning scaffolds
The characteristics of scaffolds were analyzed in terms of mechanism, functions, delivery forms, mode, numbers of scaffolds, and how to facilitate SRL by scaffolds.
The results by the mechanism of scaffolds are shown in Table 4. Twenty-one studies (72.4 %) adopted prompts or hints to scaffold SRL. Six studies (20.8 %) employed an integrated SRL tool as the mechanism of scaffolding, which was associated with the highest effect size. One study (3.4 %) used the concept map as the mechanism; another study (3.4 %) adopted the worked example as its mechanism, which was correlated with the lowest and negative effect size. Furthermore, there was no significant between-group effect among the four kinds of mechanism (Q = 4.139, df = 3, p = 0.247).
With regard to the functions of scaffolds, 19 studies (65.5 %) adopted multiple functions when designing the scaffolds. Four studies (13.8 %) preferred a metacognitive focus and another four (13.8 %) designed strategic scaffolds to support SRL. Two studies (6.9 %) adopted conceptual scaffolds. The highest effect size was associated with the strategic scaffolds, and the lowest effect size was correlated with the conceptual scaffolds (Table 5). However, there was no significant difference among the four categories (Q = 5.518, df = 3, p = 0.138).
With respect to the delivery form of scaffolds, 27 studies (93.2 %) used an indirect delivery form and one (3.4 %) used direct delivery. Only one study (3.4 %) adopted both direct and indirect delivery forms. The effect size for the direct delivery form (ES = 0.354) was lower than for indirect delivery (ES = 0.414). The highest effect size was associated with both direct and indirect delivery forms. There was significant difference among these three kinds of delivery forms (Q = 10.43, df = 2, p = 0.005) (Table 6).
In terms of mode of scaffolds, three studies (10.3 %) were coded as domain-specific scaffolds and 25 studies (86.2 %) were coded as domain-general. One study (3.4 %) was coded as using both modes. The highest effect size was associated with both modes (ES = 1.699). The lowest effect size was correlated with the domain-specific scaffolds (ES = 0.102) (Table 7). The test of heterogeneity in effect sizes indicated a significant difference between the two kinds of modes (Q = 42.77, df = 2, p = 0.000). This implies that both the domain-general and domain-specific scaffolds had a positive effect on academic performance.
Of 29 articles, 18 (62.01 %) were coded as having multiple scaffolds and 11 (37.93 %) were coded as a unique scaffold. The higher effect size was associated with the multiple kinds of scaffolds (ES = 0.577), in contrast to the unique scaffold (ES = 0.211). There was a statistically significant difference between the multiple and unique scaffolds (Q = 4.424, df = 1, p = 0.035) (Table 8). This result showed that the multiple scaffolds were more effective in improving academic performance than the unique one. This result was also corroborated by Lee et al. (2010) who provide empirical evidence that the combination of generative learning strategy prompts with metacognitive feedback can improve learners’ comprehension of science topics.
As to how to facilitate SRL by scaffolds, three types were coded for the 29 articles. Twenty-six studies (89.8 %) were coded as supporting the whole process of SRL. The remaining three studies supported monitoring, adaptation, and reflection specifically. Supporting the whole process of SRL was associated with the highest effect size (ES = 0.469) and supporting reflection with the lowest effect size (ES = −0.105). No statistically significant difference was found among the three categories (Q = 4.561, df = 3, p = 0.207) (Table 9). This result demonstrated that supporting the whole process of SRL by scaffolds was more helpful for improving academic performance than only supporting one specific phase of SRL.
Demographics of the studies
Five key demographics were examined: sample groups, sample size, learning domain, research settings, and types of computer-based learning environments. For research sample groups, three main kinds of groups were selected across the 29 articles, namely primary school (four studies), junior and senior high school (eight studies), and higher education (17 studies). The higher education group was most often targeted (58.6 %), and the primary school sample was least selected (13.8 %). This may be because it is easier for researchers working in higher education to access students engaged in higher education. The smallest effect size was associated with the primary school group (ES = 0.277), while the largest was for the junior and senior high school group (ES = 0.841). There was no significant difference between the three sample groups (Q = 2.154, df = 2, p = 0.341) (Table 10).
Slavin and Smith (2009) indicated that sample size may result in variation. Studies with small sample sizes were likely to yield larger effect sizes (Cheung and Slavin 2013). Therefore, the present study also examined the different effect sizes linked to different sample size. Ten studies (34.5 %) selected the larger sample size (101–300 participants) for greater statistical power. Thirteen studies (44.8 %) selected a medium sample size (50–100 participants), and six studies (20.7 %) selected the smaller sample size (20–50 participants). As shown in Table 11, the highest effect size was correlated with the largest sample size. No statistically significant difference among different sample sizes was found in the current study (Q = 2.709, df = 2, p = 0.258).
In terms of research learning domain, the natural science learning domain was found in 18 studies (62.1 %) and the social science domain featured in 10 (34.5 %). Only one study was set in the medical science learning domain. As shown in Table 12, the largest mean effect size was associated with natural science (ES = 0.458) and the smallest effect size was correlated with medical science (ES = 0.270). However, there was no significant difference among the three kinds of learning domain (Q = 0.186, df = 2, p = 0.911).
For the research setting variable, 19 studies (65.5 %) were conducted in the laboratory and ten studies (34.5 %) were carried out in the classroom. The higher mean effect size was associated with the classroom settings (ES = 0.757), and the lower effect size was correlated with the laboratory condition (ES = 0.265) (Table 13). No significant difference between the two kinds of research settings was found (Q = 2.655, df = 1, p = 0.103).
As to the types of computer-based learning environments, six types of environments were adopted in the 29 articles. These included hypermedia learning environment, Web-based learning environment, concept mapping tool, mobile learning environment, agent, and inquiry learning environment. The Web-based learning environment (15 studies, 51.72 %) was most used, and the mobile and inquiry learning environments were the least adopted (one study each). Six studies adopted a hypermedia learning environment: four focused on agents, and two used a concept mapping tool. The greatest effect size was associated with the Web-based learning environment (ES = 0.676) (Table 14). This suggested that scaffolding in a Web-based learning environment is more effective than for other learning environments. The mobile devices and inquiry learning environment had a negative effect size. Significant differences were found among the six categories of learning environments (Q = 19.21, df = 5, p = 0.002).
Research methodological features
Four key methodological features were examined: the research methods, type of research design, type of organization for treatment, and duration of treatment.
Table 15 shows 29 studies adopted the quantitative methods to analyze the data. All of the studies selected the quantitative methods because they offered descriptions of the empirical investigations of the effect of scaffolds via statistical techniques. The effect size achieved was 0.438.
For research design, 22 studies (75.9 %) adopted a true experimental design and seven studies (24.1 %) were the quasi-experimental. Table 16 shows the effect size for the quasi-experimental design (ES = 0.486) was higher than for the true experimental design (ES = 0.419). However, the result of the heterogeneity test revealed that there was no significant difference between the true experimental design and the quasi-experimental design (Q = 0.098, df = 1, p = 0.754).
In terms of how participants were organized for treatment, 24 studies (82.8 %) implemented scaffolds individually and five studies (17.2 %) adopted scaffolds in groups. As indicated in Table 17, a higher effect size was associated with the individual treatment (ES = 0.496) and a lower effect size was associated with groups. Moreover, a significant difference was found between the two types of organization type (Q = 7.079, df = 1, p = 0.008).
The duration of treatment is crucial in a meta-analysis. Table 18 presents the results regarding different durations for treatment. Seven studies (24.1 %) were coded as 2–4 weeks, associated with the highest effect size (ES = 0.776). One study (3.4 %) was coded as 9–24 weeks, associated with the lowest and negative effect size (ES = −0.032). There was no significant difference among different kinds of duration (Q = 9.83, df = 5, p = 0.080).
Publication bias
To test for publication bias, the classic fail-safe N was conducted to examine whether there were numbers of studies with null or negative results that have not been discovered (Rothstein et al. 2006). As shown in Table 19, the results of this test revealed that 1315 studies with null results would be needed to nullify the overall effect size. Further examination of a funnel plot demonstrated that the studies were symmetrically distributed around the combined effect size. Therefore, no publication bias was found in the current meta-analysis.
Sensitivity analysis
Sensitivity analysis was conducted to examine outliers that may skew the overall results. The one-study removal analysis (Borenstein et al. 2011) was adopted to check for the extreme positive and negative effect sizes for the overall effect size. The results indicated that the effect sizes still fall on the 95 % confidence interval (0.229–0.648). This means that the overall effect size would not be affected if any one effect size was removed.
Discussion
The findings of this meta-analysis with a total sample size of 2648 participants indicated that SRL scaffolds in computer-based learning environments produced a medium positive effect (ES = 0.438) on academic performance. This is consistent with a previous report by Schmid et al. (2014) who found that the overall weighted average effects of technology use on achievement were significant. This result was also corroborated by a recent meta-analysis reporting a positive overall effect size for ICT interventions (Archer et al. 2014). The results of meta-analyses are more accurate and reliable than individual studies because they focus on a larger sample size (Li and Ma 2010; van Ewijk 2011). The sample size in the present study was 2648, which supports the validity and accuracy of the results.
Findings suggest that a Web-based learning environment is optimal for supporting SRL. The scaffolds should support the whole process of SRL, ranging from setting goals, making plans, and enacting strategies, to adapting metacognition. Scaffolding only one phase of SRL is not enough. This result was unsurprising because SRL is an iterative and cyclical process that involves defining tasks, setting goals and making plans, enacting tactics, and adapting metacognition (Winne 2001). Therefore, it is necessary to use scaffolds to support SRL during the whole process. Integrated SRL tools with multiple functions such as setting goals, making plans, enacting strategies, self-monitoring, and self-evaluation were found to be the most effective in promoting SRL in our study.
Similar to Ifenthaler’s (2012) report, our results also suggest that domain-general scaffolds will be more effective than domain-specific scaffolds. Since self-regulation is domain-general in nature, this is not an unexpected finding (Pintrich 2000). Scaffolds that can provide a wide variety of domain-general skills including setting goals, planning, monitoring, evaluating, and adaptation can help learners to be more self-regulated and give them greater autonomy (Koedinger and Aleven 2007). However, when used in combination, both domain-general and domain-specific scaffolds had the highest effect size. Therefore, both types of scaffolds should be designed and integrated into interventions. Additionally, the combination of different kinds of scaffolds such as prompts and feedback can achieve better academic performance. It should be noted that SRL has the contextual characteristics that include the characteristics of learners, the learning environment, and their interactions (Efklides 2011). Therefore, the design of scaffolds should also consider these contextual factors to foster the skills of SRL.
The present study indicated that there were few adaptive scaffolds used to promote SRL. Adaptive scaffolding may be more beneficial for fostering SRL and lead to significant learning gains since it adjusts based on students’ learning needs (Azevedo et al. 2004, 2005). Previous studies have provided empirical evidence to support adaptive scaffolding in different learning domains, such as algebra (Kramarski and Hirsch 2003), biology (Azevedo et al. 2004), and history (Poitras and Lajoie 2014). Adaptive scaffolds can flexibly fade based on the ongoing diagnosis and evaluation of students’ knowledge and skills so that students ultimately internalize the learning process by themselves (Azevedo and Hadwin 2005).
In addition to the overall findings, the current study investigated the moderate effect on academic performance by considering the demographics of the selected studies as well as methodological features. Key findings from the meta-analysis and the way in which these impact student academic performance are now discussed.
First, although the meta-analysis indicated that no significant difference in the effect size of different research sample groups, sample sizes, learning domains, and research settings, our findings revealed that the larger sample size (101–300) can produce a larger effect size. This result was consistent with Chang and Lin (2013) who reported that the greatest mean effect size was correlated with studies with the largest sample size (101–300). This is logical because statistical power is enhanced when a sample size increases. Therefore, it is suggested that researchers should adopt the largest sample size possible so as to obtain significant statistical power and reliable results.
Second, the study revealed that the effect size of the quasi-experimental design was higher than for the true experimental design. This finding was consistent with previous studies (Cheung and Slavin 2013). Since the results of quasi-experimental studies have larger variances (Niemiec et al. 1987), more studies with true experimental designs should be adopted in future.
Third, the findings also indicated that the effect size was the highest when the treatment continued for 2–4 weeks. If the treatment lasted too long, for example 9–24 weeks, the effect size decreased dramatically. This result was supported by Clark (1983) who believed that effect size decreased when treatment continued for a long period of time.
The current study was constrained by several limitations. First, the meta-analysis only focused on measures of academic performance. It would be very useful to assess other outcomes arising from SRL scaffolds. Future studies need to examine other outcome variables, including self-regulated skills, motivations, and emotions. Second, the study only identified three main categories of moderators, namely the scaffold characteristics, demographics, and research methodologies of the selected studies. Ongoing research is needed to investigate other potential moderators to understand how to maximize academic performance when scaffolding SRL in a computer-based environment. Finally, only empirical studies were included in the present analysis. There is much to be learned from qualitative research. Future studies will analyze qualitative research to add further to our understanding of the effects of scaffolds.
Conclusion
The overall conclusion emerging from the present meta-analysis is that SRL scaffolds in a computer-based environment have positively effects on academic performance. The characteristics of scaffolds and the demographics and methodological features of studies were examined as moderating variables. Major findings revealed that both domain-general and domain-specific scaffolds can support the entire process of SRL and positively affected academic performance. It is suggested that integrated SRL tools with multiple functions and multiple kinds of scaffolds should be considered as the optimal way to promote academic performance. In addition, large sample size and true experimental designs should be adopted to obtain optimal statistical power. To eliminate confounding variables, the duration of treatment should not be too long. The main contribution of this study lies in its exploration of moderator variables to gain insights into how scaffolding treatments and study features lead to positive effects. The study contributes to the body of knowledge about how to design scaffolds to promote SRL so as to maximize students’ learning.
References
Adeyinka, T., & Mutula, S. (2010). A proposed model for evaluating the success of WebCT course content management system. Computers in Human Behavior, 26(6), 1795–1805.
Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2010). Automated, unobtrusive, action-by-action assessment of self-regulation during learning with an intelligent tutoring system. Educational Psychologist, 45(4), 224–233.
Archer, K., Savage, R., Sanghera-Sidhu, S., Wood, E., Gottardo, A., & Chen, V. (2014). Examining the effectiveness of technology use in classrooms: A tertiary meta-analysis. Computers & Education, 78, 140–149.
Azevedo, R., Cromley, J. G., & Seibert, D. (2004). Does adaptive scaffolding facilitate students’ ability to regulate their learning with hypermedia? Contemporary Educational Psychology, 29(3), 344–370.
Azevedo, R., Cromley, J. G., Winters, F. I., Moos, D. C., & Greene, J. A. (2005). Adaptive human scaffolding facilitates adolescents’ self-regulated learning with hypermedia. Instructional Science, 33(5–6), 381–412.
Azevedo, R., & Hadwin, A. F. (2005). Scaffolding self-regulated learning and metacognition–Implications for the design of computer-based scaffolds. Instructional Science, 33(5), 367–379.
Azevedo, R., Moos, D. C., Johnson, A. M., & Chauncey, A. D. (2010). Measuring cognitive and metacognitive regulatory processes during hypermedia learning: Issues and challenges. Educational Psychologist, 45(4), 210–223.
Azevedo, R., Verona, M. E., & Cromley, J. G. (2001). Fostering students’ collaborative problem solving with RiverWeb. In J. D. Moore, C. L. Redfield, & W. L. Johnson (Eds.), Artificial intelligence in education: Al-ED in the wired and wireless future (pp. 167–175). Amsterdam: IOS Press.
Azevedo, R., Witherspoon, A., Chauncey, A., Burkett, C., & Fike, A. (2009). MetaTutor: A meta cognitive tool for enhancing self-regulated learning. In R. Pirrone, R. Azevedo, & G. Biswas (Eds.), Proceedings of the AAAI fall symposium on cognitive and metacognitive educational systems (pp. 14–19). Menlo Park, CA: Association for the Advancement of Artificial Intelligence (AAAI) Press.
Bangert-Drowns, R.L., & Rudner, L.M. (1991). Meta-analysis in educational research. Paper presented to ERIC Clearinghouse on Tests, Measurement, Evaluation, Washington, DC. (ERIC Document Reproduction Service No. ED 339748).
Bannert, M., & Reimann, P. (2012). Supporting self-regulated hypermedia learning through prompts. Instructional Science, 40(1), 193–211.
Bannert, M., Sonnenberg, C., Mengelkamp, C., & Pieger, E. (2015). Short-and long-term effects of students’ self-directed metacognitive prompts on navigation behavior and learning performance. Computers in Human Behavior, 52, 293–306.
Benz, B. F. (2008). Self-regulated learning and academic achievement: A meta-analysis. International Journal of Psychology, 43(3–4), 541.
Biswas, G., Leelawong, K., Schwartz, D., Vye, N., & The Teachable Agents Group at Vanderbilt. (2005). Learning by teaching: A new agent paradigm for educational software. Applied Artificial Intelligence, 19(3–4), 363–392.
Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2011). Introduction to meta-analysis. Chichester, UK: Wiley.
Casal, G. B., Ávila, J. A. O., Lechuga, B. M., & Zych, I. (2011). The h index of the presidents of the American Psychological Association (APA) through journal articles included in the Web of Science database. International Journal of Clinical and Health Psychology, 11(1), 95–107.
Chang, M. M., & Lin, M. C. (2013). Strategy-oriented web-based English instruction—A meta-analysis. Australasian Journal of Educational Technology, 29(2), 203–216.
Chen, C. M. (2009). Personalized E-learning system with self-regulated learning assisted mechanisms for promoting learning performance. Expert Systems with Applications, 36(5), 8816–8829.
Chen, C. M., & Huang, S. H. (2014). Web-based reading annotation system with an attention based self-regulated learning mechanism for promoting reading performance. British Journal of Educational Technology, 45(5), 959–980.
Chen, C. M., Wang, J. Y., & Chen, Y. C. (2014). Facilitating English-language reading performance by a digital reading annotation system with self-regulated learning mechanisms. Journal of Educational Technology & Society, 17(1), 102–114.
Cheung, A. C., & Slavin, R. E. (2013). The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Educational Research Review, 9, 88–113.
Chung, M. K. (2000). The development of self-regulated learning. Asia Pacific Education Review, 1(1), 55–66.
Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445–459.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Cooper, H. M. (2010). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Los Angeles, CA: Sage.
Crippen, K. J., & Earl, B. L. (2007). The impact of web-based worked examples and self-explanation on performance, problem solving, and self-efficacy. Computers & Education, 49(3), 809–821.
Dabbagh, N., & Kitsantas, A. (2005). Using web-based pedagogical tools as scaffolds for self-regulated learning. Instructional Science, 33(5–6), 513–540.
Delen, E., Liew, J., & Willson, V. (2014). Effects of interactivity and instructional scaffolding on learning: Self-regulation in online video-based environments. Computers & Education, 78, 312–320.
Devolder, A., van Braak, J., & Tondeur, J. (2012). Supporting self-regulated learning in computer-based learning environments: Systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted learning, 28(6), 557–573.
Dignath, C., Buettner, G., & Langfeldt, H. P. (2008). How can primary school students learn self-regulated learning strategies most effectively?: A meta-analysis on self-regulation training programmes. Educational Research Review, 3(2), 101–129.
Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3(3), 231–264.
Duffy, M. C., & Azevedo, R. (2015). Motivation matters: Interactions between achievement goals and agent scaffolding for self-regulated learning within an intelligent tutoring system. Computers in Human Behavior, 52, 338–348.
Efklides, A. (2011). Interactions of metacognition with motivation and affect in self-regulated learning: The MASRL model. Educational Psychologist, 46(1), 6–25.
El Saadawi, G. M., Azevedo, R., Castine, M., Payne, V., Medvedeva, O., Tseytlin, E., & Crowley, R. S. (2010). Factors affecting feeling-of-knowing in a medical intelligent tutoring system: The role of immediate feedback as a metacognitive scaffold. Advances in Health Sciences Education, 15(1), 9–30.
Hacker, D., Dunlosky, J., & Graesser, A. (2009). Handbook of metacognition in education. New York, NY: Routledge.
Hannafin, M., Land, S. M., & Oliver, K. (1999). Open learning environments: foundations, methods, and models. In C. M. Reigeluth (Ed.), Instructional design theories and models (pp. 115–140). Mahwah, NJ: Lawrence Erlbaum Associates.
Hartley, K., & Bendixen, L. D. (2001). Educational research in the Internet age: Examining the role of individual characteristics. Educational researcher, 30(9), 22–26.
Hattie, J. A. (2009). Visible learning. A synthesis of over 800 meta-analyses relating to achievement. New York: Routledge.
Hattie, J., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of educational research, 66(2), 99–136.
Hedges, L. V., & Olkin, I. (1986). Statistical methods for meta-analysis. San Diego, CA: Academic.
Hogan, K., & Pressley, M. (1997). Scaffolding student learning: Instructional approaches and issues. Cambridge, MA: Brookline Books.
Ifenthaler, D. (2012). Determining the effectiveness of prompts for self-regulated learning in problem-solving scenarios. Journal of Educational Technology & Society, 15(1), 38–52.
Isaacson, R., & Fujita, F. (2012). Metacognitive knowledge monitoring and self-regulated learning. Journal of the Scholarship of Teaching and Learning, 6(1), 39–55.
Jackson, S. L., Stratford, S. J., Krajcik, J., & Soloway, E. (1994). Making dynamic modeling accessible to precollege science students. Interactive Learning Environments, 4(3), 233–257.
Karadağ, E., Bektaş, F., Çoğaltay, N., & Yalçın, M. (2015). The effect of educational leadership on students’ achievement: A meta-analysis study. Asia Pacific Education Review, 16(1), 79–93.
Kauffman, D. F., Zhao, R., & Yang, Y. S. (2011). Effects of online note taking formats and self-monitoring prompts on learning from online text: Using technology to enhance self-regulated learning. Contemporary Educational Psychology, 36(4), 313–322.
Kim, H. J., & Pedersen, S. (2011). Advancing young adolescents’ hypothesis-development performance in a computer-supported and problem-based learning environment. Computers & Education, 57(2), 1780–1789.
Kinshuk, H. W. A., Sampson, D., & Chen, N.-S. (2013). Trends in educational technology through the lens of the highly cited articles Published in the Journal of Educational Technology and Society. Educational Technology & Society, 16(2), 3–20.
Kistner, S., Rakoczy, K., Otto, B., Dignath-van Ewijk, C., Büttner, G., & Klieme, E. (2010). Promotion of self-regulated learning in classrooms: Investigating frequency, quality, and consequences for student performance. Metacognition and learning, 5(2), 157–171.
Koedinger, K. R., & Aleven, V. (2007). Exploring the assistance dilemma in experiments with cognitive tutors. Educational Psychology Review, 19(3), 239–264.
Kondo, M., Ishikawa, Y., Smith, C., Sakamoto, K., Shimomura, H., & Wada, N. (2012). Mobile assisted language learning in university EFL courses in Japan: Developing attitudes and skills for self-regulated learning. The Journal of EUROCALL, 24(02), 169–187.
Kramarski, B., & Hirsch, C. (2003). Using computer algebra systems in mathematical classrooms. Journal of Computer Assisted learning, 19(1), 35–45.
Kramarski, B., & Mizrachi, N. (2006). Online discussion and self-regulated learning: Effects of instructional methods on mathematical literacy. The Journal of Educational Research, 99(4), 218–231.
Kulik, J. A., & Kulik, C. L. C. (1989). The concept of meta-analysis. International Journal of educational research, 13(3), 227–340.
Lajoie, S. P., & Azevedo, R. (2006). Teaching and learning in technology-rich environments. In P. A. Alexander & P. H. Winne (Eds.), Handbook of educational psychology (2nd ed., pp. 803–821). Mahwah, NJ: Erlbaum.
Lajoie, S. P., Guerrera, C., Munsie, S. D., & Lavigne, N. (2001). Constructing knowledge in the context of BioWorld. Instructional Science, 29(2), 155–186.
Lee, H. W., Lim, K. Y., & Grabowski, B. L. (2010). Improving self-regulation, learning strategy use, and achievement with metacognitive feedback. Educational Technology Research and Development, 58(6), 629–648.
Lehmann, T., Hähnlein, I., & Ifenthaler, D. (2014). Cognitive, metacognitive and motivational perspectives on preflection in self-regulated online learning. Computers in Human Behavior, 32, 313–323.
Li, Q., & Ma, X. (2010). A meta-analysis of the effects of computer technology on school students’ mathematics learning. Educational Psychology Review, 22(3), 215–243.
Manlove, S., Lazonder, A. W., & de Jong, T. (2007). Software scaffolds to promote regulation during scientific inquiry learning. Metacognition and Learning, 2(2–3), 141–155.
Marée, T. J., van Bruggen, J. M., & Jochems, W. M. (2013). Effective self-regulated science learning through multimedia-enriched skeleton concept maps. Research in Science & Technological Education, 31(1), 16–30.
Molenaar, I., Roda, C., van Boxtel, C., & Sleegers, P. (2012). Dynamic scaffolding of socially regulated learning in a computer-based learning environment. Computers & Education, 59(2), 515–523.
Molenaar, I., van Boxtel, C. A., & Sleegers, P. J. (2011). Metacognitive scaffolding in an innovative learning arrangement. Instructional Science, 39(6), 785–803.
Moos, D. C., & Azevedo, R. (2008). Exploring the fluctuation of motivation and use of self-regulatory processes during learning with hypermedia. Instructional Science, 36(3), 203–231.
Narciss, S., Proske, A., & Koerndle, H. (2007). Promoting self-regulated learning in web-based learning environments. Computers in Human Behavior, 23(3), 1126–1144.
Niemiec, R. P., Samson, G., Weinstein, T., & Walberg, H. J. (1987). The effects of computer based instruction in elementary schools: A quantitative synthesis. Journal of Research on Computing in Education, 20(2), 85–103.
Nückles, M., Hübner, S., & Renkl, A. (2009). Enhancing self-regulated learning by writing learning protocols. Learning and instruction, 19(3), 259–271.
Paris, S. G., & Paris, A. H. (2001). Classroom applications of research on self-regulated learning. Educational psychologist, 36(2), 89–101.
Perry, N. E., & Winne, P. H. (2006). Learning from learning kits: gStudy traces of students’ self-regulated engagements with computerized content. Educational Psychology Review, 18(3), 211–228.
Pintrich, P. R. (2000). The role of goal orientation in self-regulated learning. In M. Boekaerts, P. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 451–502). San Diego, CA: Academic Press.
Poitras, E. G., & Lajoie, S. P. (2014). Developing an agent-based adaptive system for scaffolding self-regulated inquiry learning in history education. Educational Technology Research and Development, 62(3), 335–366.
Pressley, M., & Hilden, K. (2006). Cognitive strategies. In D. Kuhn & R. S. Siegler (Eds.), Handbook of child psychology: Volume 2: Cognition, perception, and language (6th ed., pp. 511–556). Hoboken, NJ: Wiley.
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2006). Publication bias in meta-analysis: Prevention, assessment and adjustments. UK: John Wiley & Sons.
Saye, J. W., & Brush, T. (2002). Scaffolding critical reasoning about history and social issues in multimedia-supported learning environments. Educational Technology Research and Development, 50(3), 77–96.
Schmid, R. F., Bernard, R. M., Borokhovski, E., Tamim, R. M., Abrami, P. C., Surkes, M. A., et al. (2014). The effects of technology use in postsecondary education: A meta-analysis of classroom applications. Computers & Education, 72, 271–291.
Shapiro, A. M. (2008). Hypermedia design as learner scaffolding. Educational Technology Research and Development, 56(1), 29–44.
Simons, K. D., & Klein, J. D. (2007). The impact of scaffolding and student achievement levels in a problem-based learning environment. Instructional Science, 35(1), 41–72.
Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137(3), 421.
Slavin, R. E., & Smith, D. (2009). Effects of sample size on effect size in systematic reviews in education. Educational Evaluation and Policy Analysis, 31(4), 500–506.
Stahl, E., & Bromme, R. (2009). Not everybody needs help to seek help: Surprising effects of metacognitive instructions to foster help-seeking in an online-learning environment. Computers & Education, 53(4), 1020–1028.
van den Boom, G., Paas, F., Van Merrienboer, J. J., & Van Gog, T. (2004). Reflection prompts and tutor feedback in a web-based learning environment: Effects on students’ self-regulated learning competence. Computers in Human Behavior, 20(4), 551–567.
van Ewijk, C. D. (2011). Assessing students’ acquisition of self-regulated learning skills using meta-analysis. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 376–390). Milton Park: Taylor & Francis.
Veenman, M., Van Hout-Wolters, B., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition and Learning, 1, 3–14.
Wang, T. H. (2011). Developing Web-based assessment strategies for facilitating junior high school students to perform self-regulated learning in an e-Learning environment. Computers & Education, 57(2), 1801–1812.
Wang, C.-Y., Wu, H.-K., Lee, S. W.-Y., Hwang, F.-K., Chang, H.-Y., Wu, Y.-T., et al. (2014). A review of research on technology-assisted school science laboratories. Educational Technology & Society, 17(2), 307–320.
White, B. Y., Shimoda, T. A., & Frederiksen, J. R. (2000). Facilitating students inquiry learning and metacognitive development through modifiable software advisers. In S. P. Lajoie (Ed.), Computers as cognitive tools II: No more walls: Theory change, paradigm shifts and their influence on the use of computers for instructional purposes (pp. 97–132). Mawah, NJ: Erlbaum.
Winne, P. H. (2001). Self-regulated learning viewed from models of information processing. In B. Zimmerman & D. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (pp. 153–189). Mahwah, NJ: Lawrence Erlbaum Associates Inc.
Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker & J. Dunlosky (Eds.), Metacognition in educational theory and practice. The Educational Psychology Series (pp. 277–304). Mahwah, NJ: Erlbaum.
Winters, F. I., Greene, J. A., & Costich, C. M. (2008). Self-regulation of learning within computer-based learning environments: A critical analysis. Educational Psychology Review, 20(4), 429–444.
Zhang, W. X., Hsu, Y. S., Wang, C. Y., & Ho, Y. T. (2015). Exploring the impacts of cognitive and metacognitive prompting on students’ scientific inquiry practices within an e-learning environment. International Journal of Science Education, 37(3), 529–553.
Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45(1), 166–183.
Zimmerman, B. J., & Bandura, A. (1994). Impact of self-regulatory influences on writing course attainment. American Educational Research Journal, 31(4), 845–862.
Acknowledgments
This study was supported by the 2014 Beijing youth project of the Twelfth Five-Year Plan for Educational Science “The empirical study on improving the self-regulated learning abilities in smart learning environment” (CJA14185).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zheng, L. The effectiveness of self-regulated learning scaffolds on academic performance in computer-based learning environments: a meta-analysis. Asia Pacific Educ. Rev. 17, 187–202 (2016). https://doi.org/10.1007/s12564-016-9426-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12564-016-9426-9