Abstract
Even though context, content, and task factors are considered essential parts of digitally delivered instructional feedback, their effects on learning performance are most often studied separately. A meta-analysis was carried out to address the effects of context, content, and task factors of digitally delivered instructional feedback on learning performance. A summary effect of .41 (SE = .05) was found for 116 interventions. Significant influencers on learning performance in the sample were feedback focus, discipline, assessment type, and learner control (amongst others), showing the potential to explain the heterogeneity in effects. Also, any simple feedback (e.g. verification or knowledge of result) is more effective than no feedback. A meta-regression with all significant influencers and feedback focused on the process explained the heterogeneity in learning performance.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Instructional feedback consists of evaluations that (dis)confirm a learner’s performance and/or provide additional information to adapt a learner’s behaviour and learning performance (Butler & Winne, 1995), and frequently results in improved performance (Hattie & Timperley, 2007; Winstone et al., 2017a). In addition, studies on instructional feedback typically generate guidelines to enhance the effectiveness of feedback in face-to-face situations, but the question remains whether these guidelines can be transferred to other feedback situations such as, for example, when feedback is delivered digitally (Van der Kleij et al., 2015). Digitally delivered instructional feedback appeals to a different set of interactions within and between relevant factors (e.g. feedback timing and assessment type) as well as psychological and logistical barriers. In the case of digitally delivered feedback, personal factors of the source essential in face-to-face settings are replaced by a fixed set of characteristics of the digital medium (Winstone et al., 2017a; Wu et al., 2019). To examine the effects of digitally delivered instructional feedback on learning performance, studies can be described in terms of factors that are considered to influence performance. Examples of these factors are context factors, such as the educational sector; content factors, such as the feedback message itself (e.g. feedback function); and task factors, such as the assignment or discipline in which the assessment was administered (Narciss, 2013). However, so far studies on digitally delivered instructional feedback have mainly focused either solely on context factors, content factors, or task factors—without considering interactions within and between factors. For example, Debuse et al. (2007) examined manual, semi-automated, and automated feedback to optimize the grading process, and Mandernach (2005) investigated various levels of computerized feedback, such as no feedback, knowledge of result, and knowledge of correct result. Such an approach can only provide general conclusions about the effectiveness of the cluster of interest (context, content, or task factors) whilst simultaneously contributing to an unidimensional view of the processing of instructional feedback and its influence on learning performance. As a result, transfer to other learning contexts may fail to occur as all three clusters of factors combined constitute a learning environment. Hence, the present meta-analysis investigates the impact of context, content, and task factors on digitally delivered instructional feedback and subsequent learning performance as well as the interaction between factors.
Digitally delivered instructional feedback
Commonly, instructional feedback is described as confirming the learner’s performance and/or providing corrections, or suggestions for adapting performances. This general description of feedback is adopted by many researchers, albeit in different wordings, and varying in their foci and degree of specificity (e.g. Butler & Winne, 1995; Hattie & Timperley, 2007; Narciss, 2013). One of the most detailed models of instructional feedback is proposed by Narciss (2013, p. 10). This model distinguishes two feedback loops: the internal feedback loop—focused on feedback generated by the learner—as part of which the learner assesses his or her own performance or current level of comprehension according to internally defined standards (internal assessment), and the external feedback loop—focused on feedback provided by an external feedback source, such as teachers, peers, or digital systems—as part of which the learner’s performance or current level of understanding is compared to externally defined standards (external assessment; Clark, 2012; Debuse et al., 2007). The feedback processing in both loops is influenced by context, content, and task factors (see Butler et al., 2013; Gordijn & Nijhof, 2002; Hattie & Timperley, 2007; Kluger & DeNisi, 1996; Timms et al., 2016). These factors—and interactions between them—may invoke different control actions by the learner (Narciss, 2013). Even though various terminologies are used for digital systems, each with their own distinct features, they can all be considered technical solutions for supporting learning, teaching, and delivering feedback (Suhonen, 2005). Henceforth, the term digital learning environments (DLEs; Debuse et al., 2013) will be used to refer to any technical solution (typically digital systems) used for delivering feedback. DLEs offer unique possibilities compared to their face-to-face and/or pen-and-paper counterpart, such as learner-controlled frequency, timing of feedback, and feedback display (Cheung & Slavin, 2012; Mandernach, 2005). In addition, when learners work in a DLE, they interact with a relatively fixed set of factors as programmed by the DLE designers, compared to a variety of personal factors of both the feedback source and the feedback recipient in face-to-face instructional feedback.
Instructional feedback and learning performance
If the feedback is adequate—as a result of the comparisons of the internal standards, internal feedback, external standards, and external feedback—the discrepancy between the current and desired level of performance and/or comprehension will be reduced (Narciss, 2013; Orsmond et al., 2005; Price et al., 2011). Reducing this discrepancy might positively affect learning performance, that is, research has shown mixed results regarding the influence of (digitally delivered) instructional feedback. Nevertheless, the possibility that (digitally) delivering instructional feedback can enhance learning performance is widely acknowledged (Hattie & Timperley, 2007; Narciss, 2013; Nicol & MacFarlane-Dick, 2006; Winstone et al., 2017a). Furthermore, the effect of (digitally delivered) instructional feedback is increased when providing relevant information to correct ineffective strategies that target the knowledge and/or skills relevant for successful task performance (Narciss, 2013). Furthermore, the necessity and suitability of the feedback and the assignment also determine to what extent the feedback is implemented. For example, a task that requires learners to link different information sources requires different feedback than a memorization task (Kluger & DeNisi, 1996). In a similar vein, when a learner understands the task and does not face any difficulties, feedback is often seen as unnecessary and less likely to be used (Winstone et al., 2017b).
Instructional feedback factors in the present meta-analysis
Thus far, feedback studies typically include either context, content, or task factors to determine their effectiveness for enhancing learning performance. However, this leads to an incomplete and simplistic rendering of feedback situations because context, content, and task factors are typically studied in isolation despite the fact that these factors—including the numerous interactions between factors—constitute the learning environment. Alongside this simplistic rendering, feedback terminology can vary. For example, feedback can be referred to as help-seeking, feedback-seeking, support, or feedforward, depending on the conceptual framework (see Boud et al., 2018; Panadero et al., 2016; Tanaka et al., 2002). This lack of consensus in terminology and labelling, for example, concerning definitions of immediate and delayed feedback, makes it difficult to fully cover the complexity of influential feedback factors as well as their effects on learning performance.
The implementation of external feedback is determined by various factors, such as the assessment, the timing of feedback, and the amount of information or detail of that feedback (Gordijn & Nijhof, 2002; Price et al., 2010). In addition, these factors can be fit into broader clusters of factors relating to context, content, and task factors (e.g. Butler et al., 2013; Gordijn & Nijhof, 2002; Kluger & DeNisi, 1996). As context, content, and task factors are easier to manipulate and to set as relatively fixed in DLE’s, these clusters of factors were chosen as the focus of the present meta-analysis. More specifically, context factors are related to the source of feedback and create the setting in which the learning task is administered (see e.g. Gordijn & Nijhof, 2002; Narciss, 2013). Context factors include aspects such as learner control, educational sector, rewards, study setting, and timing. The second cluster comprises content factors, including the feedback message, feedback form, feedback function, and feedback focus (see e.g. Butler et al., 2013; Hattie & Timperley, 2007). The third cluster is related to whether the learner will be provided with instructional feedback as part of the assessment approach and consists of factors that are embedded in the learning setting (Timms et al., 2016). Examples are the type of assessment, the discipline in which the assessment is conducted, the assessment developers (i.e. who developed the assessment), and the feedback display. The interaction between factors within these clusters can be illustrated when a student completes mathematical equations, followed by mandatory immediate verification feedback (typically referred to as ‘Knowledge of Result’; KR) on what they did. This example links the amount of learner control with the feedback timing, feedback form, and type of assessment.
Research aim
This meta-analysis aims (a) to synthesize research on context, content, and task factors of digitally delivered instructional feedback and their effect on learning performance, and (b) to identify which context, content, and/or task factors of digitally provided instructional feedback in DLEs—directed towards children, adolescents, and (young) adults covering primary, secondary, and higher education—are most effective in improving learning performance. More specifically, the following research questions (RQs) are addressed:
- RQ1::
-
Which context factors (educational sector, feedback timing, learner control, rewards, and study setting) moderate digitally delivered instructional feedback and are most effective in improving learning performance in digital learning environments (DLEs)?
- RQ2::
-
Which content factors (form, focus, and function) moderate digitally delivered instructional feedback and are most effective in improving learning performance in DLEs?
- RQ3::
-
Which task factors (assessment developers, assessment type, discipline, and feedback display) moderate digitally delivered instructional feedback and are most effective in improving learning performance in DLEs?
- RQ4::
-
Which combination of context, content, and task factors moderates digitally delivered instructional feedback and contributes to learning performance in DLEs?
Methods
Search strategy
Peer-reviewed articles published in English, published between January 2004 and January 2019, were obtained from the search databases ERIC, PsycINFO, and SocINDEX. Relevant search terms for (a) feedback included ‘feedback’, ‘feedback-seeking’, ‘help-seeking’, and ‘support’, and (b) digital learning environment (DLE) included ‘digital (learning) environment’, ‘digital education’, ‘computer learning’, ‘learning environment’, ‘intelligent tutoring’, and ‘hypermedia’.
Inclusion criteria
In order to be included in the meta-analysis, studies had to meet the following criteria:
-
1.
Feedback was implemented in a digital learning environment (as a tool or as a provider) involving an individual learning task.
-
2.
Quantitative outcomes were reported for individual performance (these could focus on the academic, social, or psychomotor domain).
-
3.
A control group or comparison group should be included to enable interpretation of the results of the experimental group. Furthermore, a pretest had to be included to correct for initial differences between the experimental and control group(s), or random assignment to conditions was part of the procedure.
-
4.
The participants needed to be aged 5 years or older.
The following exclusion criteria were used to narrow the scope:
-
1.
Studies involving special needs education (e.g. dyslexia, deaf/blindness, and/or attention deficits).
-
2.
Meta-analyses or systematic/literature reviews.
-
3.
Studies primarily focused on user or evaluation feedback.
-
4.
Studies primarily focused on feedback towards teachers.
-
5.
Studies primarily focused on the development of a digital learning environment.
-
6.
Studies focused on workplace learning.
-
7.
Studies in which researchers were the participants.
The search provided 4855 hits, and during the first phase, all abstracts were scanned for the inclusion/exclusion criteria. The abstracts mentioned feedback by either mentioning the word feedback or with a similar description (e.g. help or support) were scanned more thoroughly. Authors from four non-accessible articles were e-mailed with a request for a copy of their article, of whom one complied. After removing the duplicates from the sample and applying the inclusion/exclusion criteria, 81 articles were included for further analysis (see Fig. 1).
Coding of variables
The statistical quality of the articles to be included in the meta-analysis was determined through the selection process displayed in Fig. 1, extended with the criteria for exclusion. A final decision about inclusion/exclusion as a result of a lack of statistical quality always followed after a discussion between the two coders (i.e. first author and research assistant). In total, 46 articles met the inclusion criteria and were of sufficient statistical quality. In several articles, multiple interventions were included. As such, we were able to include the results of 116 interventions in our meta-analysis.
The coding of the characteristics of the feedback message started with mapping out when and how it could influence the processing of feedback, marking this as an inductive approach. A deductive approach was used to complement the information gathered by taking individual learners—and their processing of feedback—as a reference point (e.g. see Timms et al., 2016). This approach, in combination with existing research, served as a guideline for the distinction into three clusters in this meta-analysis: context factors, content factors, and task factors (cf. Butler et al., 2013; Gordijn & Nijhof, 2002; Hattie & Timperley, 2007; Kluger & DeNisi, 1996; see also Supplementary Materials Part 1 for an elaborated description of the feedback clusters).
Interrater reliability (IRR) was first determined for the general information (authors, title, publication years, sample size for the control group, and sample sizes for the experimental groups), followed by the reliability for the clusters of feedback factors. Krippendorff’s alpha for the coding of the general information (κ = 0.94) and the clusters of feedback factors (κ = 0.94) were satisfactory (McHugh, 2012).
Analysis
Comprehensive Meta-Analysis software package (CMA, version two; Biostat; see: www.meta-analysis.com) and Hierarchical Linear Modelling (HLM, version six; Raudenbush et al., 2004) were used for the data analysis. First, effect sizes and variances for each intervention were calculated. This information was then combined to compute a weighted summary effect and to perform moderator analyses to examine which factors influence the summary effect.
Effect size for interventions
CMA computed an effect size and variance for each intervention, based on the statistical information provided in the articles. Hedges’ g was selected as the effect size parameter given the concerns about bias and corrections of effect sizes (Cumming, 2012). CMA requires a pre–post correlation score when the effect size is calculated, and with the lack of this information in the articles, we used an estimated effect size of 0.7 (similar to the pre–post correlations on achievement tests provided by Cole et al., 2011). A correction for dependency in the data was applied to articles with multiple experimental groups with comparisons with the same control group or to within-person designs. Without correction, these control groups would be over-weighted in the meta-analysis (Lipsey & Wilson, 2001). Furthermore, for some interventions, multiple measures of performance were available. In these cases, all measures were used, and CMA computed a mean effect size based on these measures. The variance of this mean effect size for an intervention was adjusted in such a way that the variance decreased slightly when multiple outcome measures were reported.
Combining the effects
CMA and HLM make use of weighted analyses. The weight attached to each intervention is the inverse of its variance. Thus, studies with large sample sizes (and thus a small variance) have more influence on the computed effect size than interventions with small sample sizes. The interventions in the meta-analysis differed in many respects, resulting in the use of a random effects model to compute the summary effect and a mixed effects model for the moderator analyses. CMA was used for all analyses, except for the meta-regression analyses with multiple moderators in the same model for which HLM was used.
Publication bias
A selection bias arises when the included articles do not represent all the research conducted about the subject matter (Ellis, 2010; Rothstein et al., 2005). A funnel plot was used to discover publication bias. When a bias is present, the plot is skewed and asymmetrical rather than equally distributed; however, the funnel plot must be carefully interpreted because asymmetry is not always the result of biases, and it only informs about the presence of a bias; not which bias is present (Ellis, 2010). Duval and Tweedie’s trim-and-fill procedure was used to determine if the sample missed interventions to provide a more symmetrical funnel plot (Duval & Tweedie, 2000).
Tower of babel bias
The Tower of Babel bias is presumably present in this meta-analysis due to the focus on articles published in English. The severity of this bias was examined by coding in which country the intervention was conducted, and in which language the intervention materials were presented.
Type-I error correction
A series of significance tests were performed, increasing the likelihood of a Type-I error: finding an effect by chance when there is actually no effect. The recommendation of Polanin (2013)—the application of the ‘False Discovery Rate’ (FDR) method of Benjami and Hochberg (1995) within each group of tests (‘timeline of significance testing’)—was used to correct for Type-I errors. The FDR method finds a balance between the chances of incorrectly rejecting the null hypothesis (Type-I error) and falsely accepting the null hypothesis (Type-II error). In accordance with the FDR procedure, the p values within each group of tests (summary effect and categorical moderator analyses) were sequenced from low to high, starting with the largest value, and determined the largest p value for which
where i is the ordered p value, m is the number of significance tests, and α is the chosen level of control. The null hypothesis of pi and smaller was rejected with α = 0.05.
Outliers and model fit
The Tukey method was used to recode outliers into respective inner fence values beyond the 25th and 75th percentiles (Lipsey, 2009). Five interventions with extremely high effect sizes were recoded to the 75th percentile to prevent an excessive influence of these interventions on the results.
The model fit of the meta-regression was determined according to the Akaike Information Criterion (AIC), with lower values indicating a better model fit (Harrer et al., 2019).
Results
Effects of feedback on learning performance
The summary effect of the 116 interventions was moderate: Hedges’ g = 0.41 (SE = 0.05; 95% CI 0.32–0.50, p < 0.001). The homogeneity statistic indicated that the variation in effects is statistically detectable (Q = 295.03; df = 115; p < 0.001), meaning that the feedback interventions do not share the same true effect size and display variability in scores. The variance of the true effect sizes was estimated at T2 = 0.13 and reflected a moderate proportion of real difference in effect sizes (I2 = 61.02). Supplementary Materials part 2 summarizes the effect sizes and characteristics from the included interventions.
Publication bias
Duval and Tweedie’s trim-and-fill method (Duval & Tweedie, 2000) revealed a publication bias on the left side of the mean in the present sample (see Fig. 2). The black dots represent the estimated missing intervention studies due to publication bias (the white dots are the observed interventions). The white diamond represents the observed summary effect, and the black diamond represents the summary effect with adjustment for publication bias.
The estimated Hedges’ g would have been lower (Hedges’ g = 0.23)—with 25 trimmed studies—without the publication bias compared to the observed Hedges’ g of 0.41 (Cohen, 1988). However, if we had not reduced the extremely high effect sizes of some intervention studies (outlier reduction), the result of the analysis of publication bias would have been very different, with no estimated missing intervention studies. Figure 3 displays the corresponding funnel plot.
Tower of babel bias
For each intervention, the country in which the intervention took place and language in which the intervention materials were presented was coded. Roughly 60% of the included interventions were conducted in a native English-speaking country. The remaining 40% comprises interventions in German, Taiwanese, Spanish, and Dutch (see Table 1). The Tower of Babel bias did not appear to be as severe as previously assumed.
Moderator analyses
Many aspects of the intervention studies were coded. However, not all information was always provided in the articles in which the interventions were described, with missing data as a consequence. This missing data was deleted from the moderator analysis because missing data has no additional value in the analyses.
Context factors
The effect sizes of the context factors and their significance are displayed in Table 2. The between-group difference was statistically significant for education sector (Q-between = 33.55; df = 4; p ≤ 0.001), feedback timing (Q-between = 23.62, df = 2, p ≤ 0.001), learner control (Q-between = 57.39; df = 3; p < 0.001), and study setting (Q-between = 15.75; df = 2; p ≤ 0.001). A significant between-group difference means that the variation in effects between the various categories is statistically detectable. Table 2 shows that learning performance seems unaffected by whether participants received a reward or not (Q-between = 1.54; df = 1; p = 0.21).
Content factors
The effect sizes of the content factors of feedback and their significance are displayed in Table 3. The learning performance was affected by feedback form (Q-between = 14.83; df = 2; p ≤ 0.001), feedback focus (Q-between = 51.97; df = 5; p ≤ 0.001), and feedback function (Q-between = 17.97; df = 5; p ≤ 0.001). This indicates that the variation in effects is statistically detectable. Within the studies involving a feedback focus on the process, surface processing strategies provided a Hedges’ g of 0.53 (SE = 0.22; 95% CI 0.10 to 0.95; p = 0.02; k = 9) and deep processing strategies a Hedges’ g of 0.09 (SE = 0.06; 95% CI − 0.04 to 0.22; p = 0.17; k = 29). The Q-between for this difference between strategies was not significant 3.63 (df = 1; p = 0.06).
Task factors
The effect sizes of the task factors are displayed in Table 4. All factors displayed statistically detectable variation between the groups: assessment developers (Q-between = 12.68; df = 3; p = 0.01), assessment type (Q-between = 16.98; df = 6; p = 0.01), discipline (Q-between = 17.97; df = 5; p ≤ 0.001), and feedback display (Q-between = 6.73; df = 2; p = 0.04).
Control condition
The control condition within the studies could receive no feedback or some feedback (regardless of what the feedback encompasses). Hedges’ g was 0.61 (SE = 0.08; 95% CI 0.45–0.78; k = 50) for interventions compared with control conditions with no feedback, and Hedges’ g was 0.27 (SE = 0.05; 95% CI 0.16–0.37; k = 66) for interventions compared with control conditions receiving some feedback. The difference was significant (Q-between = 12.88; df = 1; p < 0.001).
Type-I correction
A series of significance tests were performed, resulting in a significant summary effect and significant between-group differences (p < 0.05). After applying the FDR method, relevant factors within clusters, i.e. learner control, educational sector, study setting, feedback focus, feedback form, feedback function, assessment developers, assessment type, discipline, and feedback display, were likely not false discoveries and reflected real differences.
Combining context, content, and task factors
A meta-regression was employed to examine the effects of multiple moderators simultaneously. According to Borenstein et al. (2009), a meta-regression model can include approximately one variable per ten interventions. According to that reference, the model could include a maximum of ten moderators. Meta-regression was employed separately for context, content, and task factors (see Part 3 in the Supplementary Materials). However, the number of dummy variables included in the analyses exceeded the maximum of ten moderators. Therefore, the reported results should be interpreted with caution. Due to the high number of variables, the power of the meta-regression remains low and provided models that are too full (see Part 3 in the Supplementary Materials). As a result, the number of variables in the model was minimized by implementing a forward regression focused on factors that could be adapted by practitioners. This practical approach was adopted to provide practitioners specific aspects to adjust instruction and/or instructional feedback besides the education provided in a DLE. The factors learner control, feedback timing (both are context factors), feedback form, feedback focus, feedback function (all content factors), and feedback display (task factor) were stepwise added to a basic model with as a covariate the variable indicating whether the control group did or did not receive some feedback. This covariate was essential to ensure sufficient quality of the data because inclusion of a control group allows comparisons between different groups within a study. In the first step of the forward regression, each model consisted of the covariate and one of the factors. The model with the covariate and feedback focus showed the best improvement in model fit (χ2 = 24; df = 6; p ≤ 0.001). In the next step of the forward regression, addition of one of the remaining factors—learner control, feedback timing, feedback form, feedback function, and feedback display—did not improve the model fit, leaving the best model fit for the basic model that includes feedback focus besides the covariate whether the control group did or did not receive some feedback (see Table 5).
Discussion
The increasing use of DLEs in education calls for a re-examination of existing literature about digitally delivered instructional feedback. Existing practices might apply guidelines derived from feedback in face-to-face settings, despite the different interactions between relevant clusters of factors (i.e. context, content, and task factors) as well as psychological and logistical barriers regarding digitally delivered instructional feedback (see Winstone et al., 2017b; Wu et al., 2019). The present meta-analysis identified the most effective context, content, and task factors of digitally provided feedback to improve learning performance in DLEs from studies published between January 2004 and January 2019. The summary effect of 0.41 (SE = 0.05) for the 116 interventions from 46 articles was moderate.
Context factors
The first focus was on context factors, such as the educational sector, feedback timing, learner control, receiving a reward, and study setting. The context factors feedback timing, learner control, educational sector, and study setting significantly improved learning performance, but this was not the case for receiving a reward.
With regard to feedback timing, both immediate and delayed feedback had significant and strong effects on improving learning performance, with delayed feedback being slightly more effective than immediate feedback. A combination of feedback timing approaches was ineffective. These findings indicate that clarity and consistency—as to whether participants receive immediate or delayed feedback—is more essential than the actual timing of the feedback. These findings are in line with the previous research on immediate feedback (Dihoff et al., 2003) and delayed feedback (Butler et al., 2007).
In general, many activities account for learner control, ranging from basic decisions about the pace of information presentation found in many DLEs (pacing) to determining the order of information units (sequencing), the contents to be processed (selection or content controlling), or how the contents should be displayed (representation control; see Scheiter & Gerjets, 2007). However, all of these control decisions, if available in the DLEs, rely on (a) whether and how the learner is able to decide, and (b) whether (s)he is able to act upon his/her decisions. With respect to learner control, most DLEs in the present meta-analysis have features that automatically deliver instructional feedback to the learners without actions from the teacher. The results of the present meta-analysis do not favour self-paced feedback above pacing decided by others (e.g. teacher and peer) and automatic feedback (with a slightly lower effect size). Although Recker and Pirolli (1992) found that learners with better self-regulated learning strategies perform better in DLEs with options for more learner control than learners who are less proficient in implementing self-regulated learning strategies, more learner control (i.e. self-paced) does not always directly pave a path to better learning performance. The effect sizes for self-paced feedback and pacing decided others (e.g. teacher and peers) were similar. However, combining learner control options did not positively contribute to learning (performance) and would be advised against based on these findings.
The effects of educational sector were significant for middle school (11–13 years), high school (14–18 years), and higher education (18 years and older, e.g. university [of applied sciences]). Interventions in primary education were not significant (i.e. p = 0.06). Although with caution, the findings generally show that digitally delivered instructional feedback is more effective when learners are older. First, primary and secondary education are considered more supportive contexts for students in terms of responsibility, structure, and guidance than higher education (Rodway-Dyer et al., 2009). Second, metacognition is a skill that develops with age (Efklides, 2008); thus, younger learners (i.e. primary and secondary education) might rely more heavily and/or solely on external feedback as opposed to older learners who are assumed to have better monitoring skills. Older learners might rely more on internal feedback or are better able to process the digitally delivered instructional feedback, either requested by them or delivered to them. Third, the curriculum in primary and secondary education is designed to teach (comparatively) basic skills, whereas in higher education, the learner oftentimes selects a focused set of courses, which is sometimes described as a specialization that fits a discipline. Fourth, research has shown that motivation is universally lacking for a range of subjects when learners progress from primary to secondary education (Graham et al., 2016).
Regarding the setting in which the study was conducted a significant difference was observed between school-based studies and laboratory-setting studies, in which laboratory-setting studies displayed a higher effect size than school-based studies.
Content factors
The second focus of this meta-analysis was on content factors, such as feedback form, feedback focus, and feedback function. Concerning the feedback form, differences were found between simple, more, and most informative forms of digitally delivered instructional feedback on learning (performance). All forms showed small (Hedges’ g = 0.21) to high (Hedges’ g = 0.64) effect sizes, with the highest effect for simple feedback (e.g. Knowledge of Result; KR) and the lowest effect for the most elaborate form (e.g. Knowledge of Correct Result or KCR with an explanation). Some of these effect sizes contrast with the previous research. Van der Kleij et al. (2015) found that simple feedback was ineffective, whereas the present sample showed a relatively high effect of simple feedback on learning performance. This difference might be explained by the type of learning outcomes considered. Whereas Van der Kleij and colleagues (2015) differentiated between lower and higher order outcomes, it was decided to avoid such a distinction in the present meta-analysis to ensure the inclusion of other learning domains such as social and psychomotor outcomes. In a similar vein, Jaehnig and Miller (2007) concluded that elaborated feedback (similar to the most elaborated feedback factor in the present meta-analysis) was more effective than knowledge of correct results (similar to the more elaborated feedback factor in the present meta-analysis). Moreover, more elaborated feedback was found to be more effective than most elaborated feedback, and simple feedback was more effective than more elaborated feedback. As elaborated feedback appeals to the degree of informativeness of the feedback, this can mean that learners might have more information to process at the same time. As a result, the processing of feedback might require more effort and time from the learner and might result in the inability of processing all feedback correctly and timely (Crews & Wilkinson, 2010). Finally, affective components might also play a role (Goetz et al., 2018), such as an emotional backwash that could inhibit further processing of the feedback (Pitt & Norton, 2017). These emotions might differ between DLEs and face-to-face settings, and, as a result, different types of feedback might evoke different emotions (Saplacan et al., 2018).
In the present sample, the feedback focus was mostly directed towards the task. A focus on the process, self-regulation, and the self were used less often; however, a focus on the process resulted in a higher effect size. Although the number of interventions leading to this effect size was quite limited, process-focused feedback (i.e. feedback focused on the main process to understand and/or complete a learning task) seems more beneficial for learning than task-focused feedback (feedback focused on understanding and/or completing the learning task), as well as the combination of task- and process-focused feedback (Hattie & Timperley, 2007; Sadler, 2010). Again, caution is warranted because the effectiveness of the feedback focus depends on the task, stressing the interaction between factors. Task-focused feedback is frequently highly specific and only applicable in nearly identical situations (Hattie & Timperley, 2007). Therefore, transfer to other tasks often fails to occur using feedback about the task (Brookhart, 2008). Despite the high effect size, the effect of self-focused feedback is inconclusive (see Hattie & Timperley, 2007). From a motivational perspective, the high effect size of self-focused feedback might be explained by the fact that learners adopt defensive or assertive attitudes regarding external feedback (i.e. by a teacher, peers, or DLE) to protect a (positive) self-image, and they become more socially aware leading to adapting behaviours regarding the seeking and processing of feedback (Linderbaum & Levy, 2010; Tuckey et al., 2002). If the threat from feedback to one’s self-esteem is considered low, it positively influences the learning outcome (Kluger & DeNisi, 1996).
All feedback functions and their combinations had a significant effect on learning performance, ranging from moderate to high effects sizes. The highest effect size (Hedges’ g = 0.88) was found for digitally delivered instructional feedback targeting metacognition; but it was based on a relatively low number of interventions (k = 7). Clustering the feedback function into overarching clusters to the function to which feedback predominantly belonged—cognition, metacognition, and motivation (in line with Narciss, 2013)—intended to provide a starting point for practitioners to deliver (digitally delivered) instructional feedback to learners. Nevertheless, feedback coded as cognition may also contain aspects of, for example, metacognition, and as a result, the distinctions are not as well-established as intended. In fact, the lack of knowledge and skills in one cluster can be compensated by knowledge and skills in another cluster; for example, better metacognitive skills can compensate for a lack of prior knowledge (see Veenman et al., 2006).
Task factors
The third focus comprised task factors, such as the assessment developers, assessment type, discipline, and feedback display. All task factors showed significant between-group differences (i.e. differences between subcategories). Regarding the factor assessment developers, these differences ranged from moderate (Hedges g = 0.48 and 0.54) to high (Hedges g = 1.00) effect sizes. The contrast between standardized tests and the assessments designed by teachers—albeit used with care due to the low number of articles—is of particular interest, because assessment is one aspect that teachers can influence or adapt. In addition, combining assessments from different developers (i.e. researcher-made and teacher-made) yielded no significant effect on learning (performance), which might be due to differences in the quality of the assessment or the goal for which the assessment was designed.
Regarding the assessment type, oral assignments yielded the highest effect size, and a combination of assignment types the lowest. However, give the relatively low number of studies (k = 6), the effect size for oral assignments can be considered an extreme (Hedges’ g = 0.99). More frequently occurring assessment types (e.g. linguistic questions, calculations, and writing assignments) also show effect sizes ranging from Hedges’ g = 0.25 for a combination of assignments up to Hedges’ g = 0.76 for calculations. The latter effect size for calculations can be explained by the relatively fixed sets of answers that are necessary, contrasted with the diversity of answers found in writing assignments (e.g. essays). In writing assignments, learners need to understand and critically review information in a particular domain, and they need to utilize language to communicate this to others (Read et al., 2001). Moreover, the level of writing skills might differ as a function of discipline and method of assessment (Bacha, 2001). Nevertheless, for practitioners, it is promising that (digitally delivered) instructional feedback appears effective with a diversity of assignments. Concerning the feedback display, a visualization yielded the highest effect size (Hedges’ g = 0.59), and a combination of a written and visual display yielded the lowest effect size (Hedges’ g = 0.25). Yet, when considered separately both feedback displays have a significant and positive effect on learning performance.
Besides assessment types, research has shown that there are disciplinary differences which might affect the effectiveness of digitally delivered instructional feedback. The present sample revealed similar effect sizes for language and arts, and science education, but a nonsignificant nihil effect size for social sciences. The nature of the tasks in science education, such as calculations and working with formulas, may require different feedback compared to tasks in social sciences where the number of valid answer options might be broader as compared to science education, making it more difficult to provide feedback as well as illustrating the crucial role of acknowledging interactions between factors.
Control condition
In interventions with a control condition, receiving some feedback (irrespective of what the feedback entailed) was more effective than receiving no feedback. However, research has also shown that some feedback is not always preferred to no feedback. For example, Fyfe et al. (2012) found that learners with little prior knowledge of the correct solution strategy benefitted from some feedback, whereas learners with some prior knowledge of the correct solution strategy benefitted from a learning situation without feedback. Yet, viewing feedback effects in a black-and-white fashion (e.g. ‘some’ feedback is better than no feedback, or feedback works vs. it does not work) paints a simplistic and unidimensional picture of the effectiveness of feedback. Practitioners might incorrectly perceive such a view as an one-size-fits-all approach, complicating and possibly thwarting positive effects of (digitally delivered) instructional feedback on learning performance.
Combining context, content, and task factors
A meta-regression was used to examine whether the combination of context, content, and task factors moderates digitally delivered instructional feedback and contributes to learning performance. The meta-regression showed that a feedback situation consisting of process-focused digitally delivered instructional feedback contributed significantly to learning performance. The other factors included in the meta-regression—i.e. learner control, feedback timing, feedback form, feedback function, and feedback display—did not contribute to the effect on learning performance. One explanation might be that the many combinations of context, content, and task factors in the feedback situation do not provide straightforward results. In other words, given that many combinations of factors potentially (positively) contribute to or hinder learning performance, there is no one-size-fits-all solution to (digitally delivered) instructional feedback, and the contribution to learning (performance) is not linear (see Timmers et al., 2013).
Limitations
Despite the exclusion of articles with an overall poor statistical quality, some articles with a sufficient overall quality still had some unclear methodological specifications (e.g. assessment developers). Furthermore, the number of interventions per factor was limited, and it should be noted that the feedback factors included in this study are still a subset of possibly relevant factors; thus, they do not fully reflect the complexity of feedback situations.
Certain feedback factors may also occur more often in combination with specific domains. For example, visual displays (e.g. graphs or diagrams) could appear more often in mathematics compared to languages. Although this might have influenced the effect sizes, the results suggest that the set of feedback factors included in the analyses contained multiple key contributors to learning performance. Despite the appeal of clusters of factors, (digitally delivered) instructional feedback cannot be easily assigned to one single cluster. In fact, feedback factors often cover multiple clusters, such as the assessment type at hand (task factor) which is nested in a domain or discipline within an educational sector (context level).
Finally, (digital) delivery of instructional feedback neither implies processing of feedback nor does it lead to enhanced learning performance per se. Most researchers assume in their studies that learners pay attention to the provided feedback, but whether they indeed do so cannot be inferred by relying solely on improvements in performance. Learners are not always skillful in the processing of feedback, and learning and performance improvement are not automatically guaranteed by the mere provision of feedback (Bevan et al., 2008; Weaver, 2006). Moreover, researchers argue that feedback training is essential to improve learners’ use of feedback (Dressler et al., 2019; Falchikov & Goldfinch, 2000). As a result, the incongruence between researchers’—hopeful—assumptions about the idealized feedback use by learners in actual practice (either in laboratory or school settings) calls for critically re-examining instructional feedback practices (Molloy & Boud, 2013) in primary education and secondary education, because relatively few interventions from these educational sectors were included in the present meta-analysis.
Future prospects
His meta-analysis showed that the effects of digitally delivered instructional feedback on learning performance are inextricably interwoven with context-, content-, and task-related factors. The results suggest that digitally delivered instructional feedback should focus on the learning process and that situations in which learners have to deal with many combinations of factors should be avoided. The results also prompt researchers to expand research into digitally delivered instructional feedback research in primary and secondary education, as these differ from higher education in terms of subjects, degree of learner control, and assessment types. Likewise, more studies are welcomed that elaborate on existing guidelines about effective (digitally delivered) instructional feedback by clarifying the contexts in which these guidelines can and should be applied and which factors contribute to learning performance. Finally, future research can elaborate on how guidelines for instructional feedback—derived from both face-to-face and digital settings—complement each other, so as to offer more contextualized guidelines to practitioners for implementing (digitally delivered) instructional feedback in their classrooms.
References
Articles marked with an asterisk indicate articles included in this meta-analysis.
*Abrami, P. C., Venkatesh, V., Meyer, E. J., & Wade, C. A. (2013). Using electronic portfolios to foster literacy and self-regulated learning skills in elementary students. Journal of Educational Psychology, 105(4), 1188–1209. https://doi.org/10.1037/a0032448
*AbuSeileek, A. F. (2013). Using track changes and word processor to provide corrective feedback to learners in writing. Journal of Computer Assisted Learning, 29, 319–333. https://doi.org/10.1111/jcal.12004
*Alemán, J. L. F., Palmer-Brown, D., & Jayne, C. (2011). Effects of response-driven feedback in computer science learning. IEEE Transactions on Education, 54(3), 501–508. https://doi.org/10.1109/TE.2010.2087761
Bacha, N. (2001). Writing evaluation: What can analytic versus holistic essay scoring tell us? System, 29, 371–383. https://doi.org/10.1016/S0346-251X(01)00025-2
*Baralt, M. (2013). The impact of cognitive complexity on feedback efficacy during online versus face-to-face interactive tasks. Studies in Second Language Acquisition, 35, 689–725. https://doi.org/10.1017/S0272263113000429
Benjami, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x
Bevan, R., Badge, J., Cann, A., Wilmott, C., & Scott, J. (2008). Seeing eye-to-eye? Staff and student views on feedback. Bioscience Education, 12(1), 1–15. https://doi.org/10.3108/beej.12.1
*Booth, J. L., Koedinger, K. R., Newton, K. J., & Lange, K. E. (2013). Differentiating instruction: Providing the right kinds of worked examples for individual students. Retrieved from Society for Research on Educational Effectiveness: https://eric.ed.gov/?id=ED563053
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Wiley.
Boud, D., Ajjawi, R., Dawson, P., & Tai, J. (2018). Creating an agenda for developing students’ evaluative judgement. In D. Boud, P. Dawson, R. Ajjawi, & J. Tai (2018), Developing evaluative judgement in higher education: Assessment for knowing and producing quality work (pp. 186–195). Routledge.
Brookhart, S. M. (2008). How to give effective feedback to your students. ASCD.
*Buder, J., & Bodemer, D. (2008). Supporting controversial CSCL discussions with augmented group awareness tools. Computer-Supported Collaborative Learning, 3, 123–139. https://doi.org/10.1007/s11412-008-9037-5
*Butchart, S., Forster, D., Gold, I., Bigelow, J., Korb, K., Oppy, G., & Serrenti, A. (2009). Improving critical thinking using web based argument mapping exercises with automated feedback. Australasian Journal of Educational Technology, 25(2), 268–291. https://doi.org/10.14742/ajet.1154
Butler, A. C., Godbole, N., & Marsh, E. J. (2013). Explanation feedback is better than correct answer feedback for promoting transfer of learning. American Psychological Association, 105(2), 290–298. https://doi.org/10.1037/a0031026
Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245–281. https://doi.org/10.3102/00346543065003245
Cheung, A. C. K., & Slavin, R. E. (2012). How features of educational technology applications affect student reading outcomes: A meta-analysis. Educational Research Review, 7, 198–215. https://doi.org/10.1016/j.edurev.2012.05.002
Clark, I. (2012). Formative assessment: Assessment is for self-regulated learning. Educational Psychology Review, 24(2), 205–249. https://doi.org/10.1007/s10648-011-9191-6
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Academic Press.
Cole, R., Haimson, J., Perez-Johnson, I., & May, H. (2011). Variability in pretest-posttest correlation coefficients by student achievement level (NCEE Reference Report 2011-4033). U.S. Department of Education, National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences.
Crews, T. B., & Wilkinson, K. (2010). Students’ perceived preference for visual and auditory assessment with e-handwritten feedback. Business Communication Quarterly, 73(4), 399–412. https://doi.org/10.1177/1080569910385566
Cumming, G. (2012). Understanding the New Statistics: Effect sizes, confidence intervals, and meta-analysis. Routledge.
Debuse, J., Lawley, M., & Shibl, R. (2007). The implementation of an automated assessment feedback and quality assurance system for ICT courses. Journal of Information Systems Education, 18(4), 491–502.
Dihoff, R. E., Brosvic, G. M., & Epstein, M. L. (2003). The role of feedback during academic testing: The delay retention effect revisited. The Psychological Record, 53, 533–548. https://doi.org/10.1007/BF03395451
Dressler, R., Chu, M.-W., Crossman, K., & Hilman, B. (2019). Quantity and quality of uptake: Examining surface and meaning-level feedback provided by peers and an instructor in a graduate research course. Assessing Writing, 39, 14–24. https://doi.org/10.1016/j.asw.2018.11.001
Duval, S., & Tweedie, R. (2000). Trim and Fill: A simple funnel-plot-based method of testing and adjusting for publication in meta-analysis. Biometrics, 56, 455–463. https://doi.org/10.1111/j.0006-341X.2000.00455.x
*Dzikovska, M., Steinhauser, N., Farrow, E., Moore, J., & Campbell, G. (2014). BEETLE II: Deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics. International Journal of Artificial Intelligence in Education, 24, 284–332. https://doi.org/10.1007/s40593-014-0017-9
Efklides, A. (2008). Metacognition. Defining its facets and levels of functioning in relation to self-regulation and co-regulation. European Psychologist, 13(4), 277–287. https://doi.org/10.1027/1016-9040.13.4.277
Ellis, P. D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge University Press.
Falchikov, N., & Goldfinch, J. (2000). Student peer-assessment in higher education: A meta-analysis comparing peer and teacher marks. Review of Educational Research, 70(3), 287–322. https://doi.org/10.3102/00346543070003287
Fyfe, E. R., Rittle-Johnson, B., & DeCaro, M. S. (2012). The effects of feedback during exploratory mathematics problem solving: Prior knowledge matters. Journal of Educational Psychology, 104(4), 1094–1108. https://doi.org/10.1037/a0028389
*Gerard, L. F., Ryoo, F., McElhaney, K. W., Liu, O. L., Rafferty, A. N., & Linn, M. C. (2016). Automated guidance for student inquiry. Journal of Educational Psychology, 108(1), 60–81. https://doi.org/10.1037/edu0000052
Goetz, T., Lipnevich, A. A., Krannich, M., & Gogol, K. (2018). Performance feedback and emotions. In A. A. Lipnevich & J. K. Smith (Eds.), The Cambridge handbook of instructional feedback (pp. 554–574). Cambridge University Press.
Gordijn, J., & Nijhof, W. J. (2002). Effects of complex feedback on computer-assisted modular instruction. Computers & Education, 39, 183–200. https://doi.org/10.1016/S0360-1315(02)00025-8
Graham, S., Courtney, L., Tonkyn, A., & Marinis, T. (2016). Motivational trajectories for early language learning across the primary-secondary school transition. British Educational Research Journal, 42(2), 682–702. https://doi.org/10.1002/berj.3230
*Hagemans, M. G., Van der Meij, H., & De Jong, T. (2013). The effects of a concept map-based support tool on simulation-based inquiry learning. Journal of Educational Psychology, 105(1), 1–24. https://doi.org/10.1037/a0029433
Harrer, M., Cuijpers, P., Furukawa, T. A, & Ebert, D. D. (2019). Doing Meta-Analysis in R: A Hands-on Guide, obtained from https://bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/. 105281/zenodo.2551803
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
*Hegele, M., & Heuer, H. (2013). Age-related variations of visuomotor adaptation result from both the acquisition and the application of explicit knowledge. Psychology and Aging, 28(2), 333–339. https://doi.org/10.1037/a0031914
*Heift, T. (2010). Prompting in CALL: A longitudinal study of learner uptake. The Modern Language Journal, 94(2), 198–216. https://doi.org/10.1111/j.1540-4781.2010.01017.x
*Henriksson, M. P., Elwin, E., & Juslin, P. (2010). Age-related variations of visuomotor adaptation result from both the acquisition and the application of explicit knowledge. Psychology and Aging, 28(2), 333–339. https://doi.org/10.1037/a0031914
*Heo, M., & Chow, A. (2005). The impact of computer augmented online learning and assessment tool. Educational Technology & Society, 8(1), 113–125.
*Hooshyar, D., Ahmad, R. B., Yousefi, M., Fathi, M., Abdollahi, A., Horng, S. J., & Lim, H. (2016). A solution-based intelligent tutoring system integrated with an online game-based formative assessment: Development and evaluation. Educational Technology Research and Development, 64(4), 787–808. https://doi.org/10.1007/s11423-016-9433-x
*Hsiao, H.-S., Chang, C.-S., Chen, C.-J., Wu, C.-H., & Lin, C.-Y. (2015). The influence of Chinese character handwriting diagnosis and remedial instruction system on learners of Chinese as a foreign language. Computer Assisted Language Learning, 28(4), 306–324. https://doi.org/10.1080/09588221.2013.818562
*Huang, C.-J., Wang, Y.-W., Huang, T.-H., Chen, Y.-C., Chen, H.-M., & Chang, S.-C. (2011). Performance evaluation of an online argumentation learning assistance agent. Computers & Education, 57, 1270–1280. https://doi.org/10.1016/j.compedu.2011.01.013
*Huang, T.-H., Liu, Y.-C., & Shiu, C.-Y. (2008). Construction of an online learning system for decimal numbers through the use of cognitive conflict strategy. Computers & Education, 50, 61–76. https://doi.org/10.1016/j.compedu.2006.03.007
*Hung, P.-H., Hwang, G.-J., Lin, Y.-F., Wu, T.-H., & Su, I.-H. (2013). Seamless connection between learning and assessment: Applying progressive learning tasks in mobile ecology inquiry. Educational Technology & Society, 16(1), 194–205.
Jaehnig, W., & Miller, M. L. (2007). Feedback types in programmed instruction: A systematic review. The Psychological Record, 57, 219–232.
Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.
*Krause, U.-M., & Stark, R. (2010). Reflection in example- and problem-based learning: Effects of reflection prompts, feedback and cooperative learning. Evaluation & Research in Education, 23(4), 255–272. https://doi.org/10.1080/09500790.2010.519024
*Lan, Y.-F., Tsai, P.-W., Yang, S.-H., & Hung, C.-L. (2012). Comparing the social knowledge construction behavioral patterns of problem-based online asynchronous discussion in e/m-learning environments. Computers & Education, 59, 1122–1135. https://doi.org/10.1016/j.compedu.2012.05.004
*Lee, C., Wong, K. C. K., Cheung, W. K., & Lee, F. S. L. (2009a). Web-based essay critiquing and EFL students’ writing: A quantitative and qualitative investigation. Computer Assisted Language Learning, 22(1), 57–72. https://doi.org/10.1080/09588220802613807
*Lee, H. W., Lim, K. Y., & Grabowski, B. (2009b). Generative learning strategies and metacognitive feedback to facilitate comprehension of complex science topics and self-regulation. Journal of Educational Multimedia and Hypermedia, 18(1), 5–25.
*Leopold, C., & Richard, M. (2015). An imagination effect in learning from scientific text. Journal of Educational Psychology, 107(1), 47–63. https://doi.org/10.1037/a0037142
Linderbaum, B. A., & Levy, P. E. (2010). The development and validation of the Feedback Orientation Scale (FOS). Journal of Management, 36, 1372–1405. https://doi.org/10.1177/0149206310373145
Lipsey, M. W. (2009). The primary factors that characterize effective interventions with juvenile offenders: A meta-analytic overview. Victims & Offenders, 4, 124–147. https://doi.org/10.1080/15564880802612573
Lipsey, M. W., & Wilson, D. B. (2001). Practical Meta-analysis. Sage Publications.
*Llorens, A. C., Vidal-Abarca, E., & Cerdán, R. (2016). Formative assessment to transfer self-regulation of task-oriented reading strategies. Journal of Computer Assisted Learning, 32, 314–331. https://doi.org/10.1111/jcal.12134
*Loewen, S., & Erlam, R. (2006). Corrective feedback in the chatroom: An experimental study. Computer Assisted Language Learning, 19(1), 1–14. https://doi.org/10.1080/09588220600803311
Mandernach, B. J. (2005). Relative effectiveness of computer-based and human feedback for enhancing student learning. The Journal of Educators Online, 2(1), 1–17.
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282.
*Mohammadzadeh, A., & Sarkhosh, M. (2018). The effects of self-regulatory learning through computer-assisted intelligent tutoring system on the improvement of the EFL learners’ speaking ability. International Journal of Instruction, 11(2), 167–184. https://doi.org/10.12973/iji.2018.11212a
Molloy, E., & Boud, D. (2013). Feedback models for learning, teaching and performance. In J. M. Spector, D. Merrill, J. Elen, & M. J. Bishop (Eds.), Handbook of research on educational communications and technology (pp. 413–424). Springer.
*Moos, D. C. (2011). Self-regulated learning externally generated feedback with hypermedia. Journal of Educational Computing Research, 44(3), 261–294. https://doi.org/10.2190/EC.44.3.b
*Moreno, R., & Mayer, R. E. (2005). Role of guidance, reflection, and interactivity in an agent-based multimedia game. Journal of Educational Psychology, 97(1), 117–128. https://doi.org/10.1037/0022-0663.97.1.117
*Moreno, R., Ozogul, G., & Reisslein, M. (2011). Teaching with concrete and abstract visual representations: Effects on students’ problem solving, problem representations, and learning perceptions. Journal of Educational Psychology, 103(1), 32–47. https://doi.org/10.1037/a0021995
Narciss, S. (2013). Designing and evaluating tutoring feedback strategies for digital environments on the basis of the Interactive Tutoring Feedback Model. Digital Education Review, 23, 7–26.
Nicol, D. J., & MacFarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090
*Nihalani, P. K., Mayrath, M., & Robinson, D. H. (2011). When feedback harms and collaboration helps in computer simulation environments: An expertise reversal effect. Journal of Educational Psychology, 103(4), 776–785. https://doi.org/10.1037/a0025276
Orsmond, P., Merry, S., & Reiling, K. (2005). Biology students’ utilization of tutors’ formative feedback: A quality interview study. Assessment & Evaluation in Higher Education, 30(4), 369–386. https://doi.org/10.1080/02602930500099177
Panadero, E., Brown, G., & Strijbos, J.-W. (2016). The future of student self-assessment: A review of known unknowns and potential directions. Educational Psychology Review, 28(4), 803–830. https://doi.org/10.1007/s10648-015-9350-2
*Penning de Vries, B., Cucchiarini, C., Bodnar, S., Strik, H., & Van Hout, R. (2015). Spoken grammar practice and feedback in an ASR-based CALL system. Computer Assisted Language Learning, 28(6), 550–576. https://doi.org/10.1080/09588221.2014.889713
Pitt, E., & Norton, L. (2017). ‘Now that’s the feedback I want!’ Students’ reactions to feedback on graded work and what they do with it. Assessment & Evaluation in Higher Education, 42(4), 499–516. https://doi.org/10.1080/02602938.2016.1142500
Polanin, J. R. (2013). Addressing the issue of meta-analysis multiplicity in Education and Psychology (Doctoral dissertation). Retrieved from https://ecommons.luc.edu/luc_diss/539/
Price, M., Handley, K., Millar, J., & O’Donovan, B. (2010). Feedback: All that effort, but what’s the effect? Assessment & Evaluation in Higher Education, 35(3), 277–289. https://doi.org/10.1080/02602930903541007
*Rau, M., Aleven, V., & Rummel, N. (2015). Successful learning with multiple graphical representations and self-explanation prompts. Journal of Educational Psychology, 107(1), 30–46. https://doi.org/10.1037/a0037211
*Rau, M. A., Aleven, V., & Rummel, N. (2016). Supporting students in making sense of connections and in becoming perceptually fluent in making connections among multiple graphical representations. Journal of Educational Psychology, 109(3), 1–18. https://doi.org/10.1037/edu0000145
Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2004). HLM 6 for Windows [Computer software]. Scientific Software International Inc.
Read, B., Francis, B., & Robson, J. (2001). Playing safe’: Undergraduate essay writing and the presentation of the student ‘voice. British Journal of Sociology of Education, 22(3), 387–399. https://doi.org/10.1080/0142569012006799
Recker, M., & Pirolli, P. (1992). Student strategies for learning programming from a computational environment. In C. Frasson, G. Gauthier, & G. I. McGalla (Eds.), Proceedings of the international conference on intelligent tutoring systems (pp. 382–394). Springer.
*Rey, D. G. (2011). Time advice and learning questions in computer simulations. Australasian Journal of Educational Technology, 27(3), 397–410. https://doi.org/10.14742/ajet.951
Rodway-Dyer, S., Dunne, E., & Newcombe, M. (2009). Audio and screen visual feedback to support student learning. In ALT-C 2009 "In dreams begins responsibility"—Choice, evidence and change, 8–10 September 2009, Manchester.
Rothstein, H. R., Sutton, A. J., & Borenstein, M. (2005). Publication bias in meta-analysis. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis. Prevention, assessment and adjustments. Wiley.
Sadler, D. R. (2010). Beyond feedback: Developing student capability in complex appraisal. Assessment & Evaluation in Higher Education, 35(5), 535–550. https://doi.org/10.1080/02602930903541015
Saplacan, D., Herstad, J., & Pajalic, Z. (2018). Feedback from digital systems used in higher education: An inquiry into triggered emotions. In G. Craddock, C. Doran, L. McNutt, & D. Rice (2018), Transforming out world through design, diversity and education (pp. 421–430). IOS Press BV.
Scheiter, K., & Gerjets, P. (2007). Learner control in hypermedia environments. Educational Psychology Review, 19, 285–307. https://doi.org/10.1007/s10648-007-9046-3
*Schoor, C., Kownatzki, S., Narciss, S., & Körndle, H. (2014). Effects of feeding back the motivation of a collaboratively learning group. Electronic Journal of Research in Educational Psychology, 12, 1696–2095. https://doi.org/10.14204/ejrep32.13077
*Schworm, S., & Gruber, H. (2012). e-Learning in universities: Supporting help-seeking processes by instructional prompts. British Journal of Educational Technology, 43(2), 272–281. https://doi.org/10.1111/j.1467-8535.2011.01176.x
*Stafford, C. A., Bowden, H. W., & Sanz, C. (2012). Optimizing language instruction: Matters of explicitness, practice, and cue learning. Language Learning, 62(3), 741–768. https://doi.org/10.1111/j.1467-9922.2011.00648.x
*Stahl, E., & Bromme, R. (2009). Not everybody needs help to seek help: Surprising effects of metacognitive instructions to foster help-seeking in an online environment. Computers & Education, 53, 1020–1028. https://doi.org/10.1016/j.compedu.2008.10.004
Suhonen, J. (2005). A formative development method for digital learning environments in sparse learning communities (Doctoral dissertation, University of Joensuu). Retrieved from ftp://www.cs.uku.fi/pub/Dissertations.
Tanaka, A., Murakami, Y., Okuno, T., & Yamauchi, H. (2002). Achievement goals, attitudes toward help seeking, and help-seeking behavior in the classroom. Learning and Individual Differences, 13(1), 23–35. https://doi.org/10.1016/S1041-6080(02)00043-2
Timmers, C., Braber-Van den Broek, J., & Van den Berg, S. M. (2013). Motivational beliefs, student effort, and feedback behaviour in computer-based formative assessment. Computers & Education, 60(1), 25–31. https://doi.org/10.1016/j.compedu.2012.07.007
Timms, M., DeVelle, S., & Lay, D. (2016). Towards a model of how learners process feedback: A deeper look at learning. Australian Journal of Education, 60(2), 128–145. https://doi.org/10.1177/0004944116652912
Tuckey, M., Brewer, N., & Williamson, P. (2002). The influence of motives and goal orientation on feedback seeking. Journal of Occupational and Organizational Psychology, 75, 195–216. https://doi.org/10.1348/09631790260098677
*Valdez, A. (2012). Computer-based feedback and goal intervention: Learning effects. Educational Technology Research and Development, 60, 769–784. https://doi.org/10.1007/s11423-012-9252-7
*Van der Kleij, F. M., Eggen, T. J. H. M., Timmers, C. F., & Veldkamp, B. P. (2012). Effects of feedback in a computer-based assessment for learning. Computers & Education, 58, 263–272. https://doi.org/10.1016/j.compedu.2011.07.020
Van der Kleij, F. M., Feskens, R. C., & Eggen, T. J. (2015). Effects of feedback in a computer-based learning environment on students’ learning outcomes: A meta-analysis. Review of Educational Research, 85(4), 475–511. https://doi.org/10.3102/0034654314564881
Veenman, M. C. J., Van Hout-Wolters, B. H. A. M., & Afflerbach, P. (2006). Metacognition and learning: Conceptual and methodological considerations. Metacognition & Learning, 1, 3–14. https://doi.org/10.1007/s11409-006-6893-0
*Walker, E., Rummel, N., & Koedinger, K. R. (2014). Adaptive intelligent support to improve peer tutoring in algebra. International Journal of Artificial Intelligence in Education, 24, 33–61. https://doi.org/10.1007/s40593-013-0001-9
Weaver, M. R. (2006). Do students value feedback? Student perceptions of tutors’ written responses. Assessment & Evaluation in Higher Education, 31(3), 379–394. https://doi.org/10.1080/02602930500353061
Winstone, N. E., Nash, R. A., Parker, M., & Rowntree, J. (2017a). Supporting learners’ agentic engagement with feedback: A systematic review and a taxonomy of recipience processes. Educational Psychologist, 52(1), 17–37. https://doi.org/10.1080/00461520.2016.1207538
Winstone, N. E., Nash, R. A., Rowntree, J., & Parker, M. (2017b). ‘It’d be useful, but I wouldn’t use it’: Barriers to university students’ feedback seeking and recipience. Studies in Higher Education, 42, 2026–2041. https://doi.org/10.1080/03075079.2015.1130032
*Wojcikowski, K., & Brownie, S. (2013). Generic reflective feedback: An effective approach to developing clinical reasoning skills. Journal of Computer Assisted Learning, 29, 371–382. https://doi.org/10.1111/jcal.12012
Wu, M., Xu, X., Kang, L., Zhao, J. L., & Liang, L. (2019). Encouraging people to embrace feedback-seeking in online learning: An investigation of informational and relational drivers. Internet Research, 29(4), 749–771. https://doi.org/10.1108/IntR-04-2017-0162
*Wu, P.-H., Hwang, G.-J., Milrad, M., Ke, H.-R., & Huang, Y.-M. (2012). An innovative concept map approach for improving students’ learning performance with an instant feedback mechanism. British Journal of Educational Technology, 43(2), 217–232. https://doi.org/10.1111/j.1467-8535.2010.01167.x
*Zeglen, E., & Rosendale, J. A. (2018). Increasing online information retention: Analyzing the effects of visual hints and feedback in educational games. Journal of Open, Flexible and Distance Learning, 22(1), 23–33.
*Zumbach, J., Reimann, P., & Koch, S. C. (2006). Monitoring students’ collaboration in computer-mediated collaborative problem-solving: Applied feedback approaches. Journal of Educational Computing Research, 35(4), 399–424. https://doi.org/10.2190/2G3G-5M86-8474-76NV
Acknowledgements
We want to thank Hidde Ozinga and Elise Voss for their help.
Funding
No funding was received for conducting this study.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Brummer, L., de Boer, H., Mouw, J.M. et al. A meta-analysis of the effects of context, content, and task factors of digitally delivered instructional feedback on learning performance. Learning Environ Res (2024). https://doi.org/10.1007/s10984-024-09501-4
Received:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10984-024-09501-4