Literacy and critical thinking are key competences for lifelong learning (European Commission, 2019). Writing a synthesis from multiple sources is a complex task with a great potential for promoting the development of such competences (Klein et al., 2016; Mateos et al., 2014). It requires students to understand the ideas contained in the texts; to select information; to organize and compare it and integrate it to establish a new structure that gives rise to a new text (Nelson & King, 2023; Segev-Miller, 2007; Spivey & King, 1989). At the college level, students have to undertake synthesis writing tasks that involve writing texts after reading from one or more sources (Klein et al., 2016; Perin, 2013) (e.g., essays, monographic papers, reports, etc.). In this context, the overall aim of this study was to examine the effectiveness of two instructional aids, a guide versus a rubric, in teaching university students the process of writing a synthesis from sources with contradictory views on a topic. In the following sections, we present the rationale for the intervention study.

Synthesis writing from sources presenting contradictory positions on a topic

When students are asked to write a synthesis from texts presenting conflicting views on a given issue, intertextual integration may contribute to develop their critical thinking and argumentative skills. It is essential for students to gain awareness of the different points of view, understand and endeavor to resolve the ensuing controversies, integrating the arguments and counterarguments put forward from each position, in order to reach a reasoned conclusion (Voss, 2001). An effective synthesis, therefore, necessarily explores in depth the various positions before offering a conclusion and reaches beyond the commonly observed tendency, known as “my side bias” (Wolfe et al., 2009), to take a stance from the beginning and merely seek points on which to support that position.

To achieve this, students may follow diverse strategies -refutation, weighing and/or synthesis/design claim (Nussbaum & Edwards, 2011; Nussbaum & Schraw, 2007). According to the above authors, students who employ the refutation strategy fundamentally defend and adopt a position, rejecting the opposing view, which they consider to be erroneous, irrelevant or ungrounded. By contrast, students applying the weighing strategy write texts acknowledging the advantages and drawbacks of each position, comparing and evaluating the pros and cons of the alternative views, as a consequence of which they adopt a stance in proportion to the weight of the arguments. The third strategy, called synthesis/design claim, likewise involves conceding the truth of the counterargument although in this case by designing a solution that will mitigate negative consequences associated with it. The writer finds an alternative solution that preserves both positions’ advantages while minimizing their negative consequences. As Nussbaum (2021) argued in a more recent work, the last two argument strategies are also refutational because they reduce the force of a counterargument but, unlike the simple or conventional refutations of counterarguments, not the truth or value of its premises or conclusion, so they can be seen as integrative refutations.

The way in which the different sources are integrated depends, among other factors, on the aim of the argumentative writing task (Nussbaum, 2021). In persuasive writing to convince and ‘win’ others, a conventional refutation of the counterargument could be an effective strategy. On the other hand, when addressing a reflective essay with the aim of learning the freedom to change positions, make concessions, and take intermediate positions by suggesting creative solutions to problems, may better facilitate conceptual change. In relation to the types of argument, Nussbaum (2021) distinguishes between practical arguments, aimed at deliberating about different courses of action in order to achieve a certain goal or value, and theoretical and scientific arguments, which are focused on establishing the truth and validity of the evidence and arguments in support of competing scientific models. A practical argument is precisely the type of argument applicable to the discussion topics that are used in our studies (e.g., whether external performance assessment is an effective measure for improving the quality of education). According to Nussbaum, design claims and weighing values are appropriate strategies in the case of practical arguments, given that generally every course of action has costs and benefits. In our work we go further by establishing a progression from argumentation based mainly on weighing to argumentation based on designing solutions. By weighing, one of the positions is assumed at its highest value with no changes or conditions attached. However, by designing solutions to solve the difficulties raised on one side of the issue, a new position is generated by synthesizing the opposing positions, but which does not correspond to any of them.

Teaching ‘how’ to write an argumentative synthesis

The difficulties met by many students when writing a synthesis from multiple sources have aroused interest in examining the effectiveness of various instructional activities (Barzilai et al., 2018; Mateos & Solé, 2009). In a review study, Van Ockenburg et al. (2019) stated that in order to acquire synthesizing skills it was necessary to involve students in a set of learning activities that promote all three transformational processes: selecting information from sources, connecting, and organizing that information. As discussed in the foregoing paragraph, the process of connecting information between texts, known as intertextual integration, is a central aspect of synthesis writing. In this regard, a review by Barzilai et al. (2018) revealed that the most frequent instructional strategies and practices to promote intertextual integration were to facilitate explicit instruction on integration, provide guidelines for the integration process, manage graphic organizers, model the integration process, engage students in collaborative discussions and practices, provide individual practice, and give feedback to students. Studies on interventions with university students for teaching how to write argumentative texts also show the effectiveness of explicit instruction (e.g. MacArthur et al., 2015; Song & Ferretti, 2013) and of graphic organizers (Kopp & Mandl, 2011; Nussbaum, 2008; Van Amelsvoort et al., 2007), especially when these are accompanied by critical questions that must be asked before writing an integrating paragraph that prompts a judgment on the merits of certain forms of argumentation (Nussbaum & Edwards, 2011).

In line with these approaches, in earlier research with university students several studies proved the effectiveness of some of those instructional practices for learning ‘how’ to write an argumentative synthesis from sources presenting conflicting views on a topic. The instructional activities were explicit instruction on the argumentative synthesis writing process through explanation and video-modelling followed by practice sessions either in collaboration with a partner or individually supported by a written guide. The guide contained a graphic organizer for selecting, comparing and prioritizing the arguments put forward by different sources and questions to prompt the use of integration strategies, by adapting the critical questions used by Nussbaum and Edwards (2011), as well as the steps in the writing process (planning, textualizing and revising). This comprehensive intervention improved the ability to select arguments from conflicting sources and to integrate them by weighing or synthesizing to a greater extent than practice with the support of the written guide but without explicit instruction (Mateos et al., 2018). The intervention based on the explicit strategy instruction and practice supported by the guide also proved to be more effective than a traditional intervention focused on formal features of argumentative synthesis texts (Luna et al., 2023).

Teaching synthesis writing assessment criteria

The aforementioned review by Barzilai et al. (2018) also revealed that the focus of most interventions designed to promote the integration of information from multiple sources is centered on teaching ‘how’ to integrate, providing different forms of instruction in the processes or strategies to follow, whereas only a few have included integration assessment criteria, i.e. what is a good ‘outcome’, as a component of the interventions. In order to learn quality assessment criteria, in these intervention studies students discussed examples of good and poor synthesis texts (Boscolo et al., 2007), self-assessed their written synthesis according to a set of explicit assessment criteria (Segev-Miller, 2004) or instructors provided feedback to students using analytic score rubrics (Zhang, 2013). Moreover, these few studies including synthesis assessment criteria were focused on integrating complementary rather than contradictory sources. Furthermore, in all these cases the quality criteria have been used to assess the written products but were not given in advance of the writing task to help students understand the aims of the task and to guide them in the writing process. An instructional rubric could be an efficient aid in this direction.

Instructional rubrics as aids to the writing process

A rubric is a rating guide providing explicit criteria for judging the quality of a written work, as well as the different levels of performance associated to each criterion (Andrade et al., 2010). Rubrics promote the objectivity, validity and dependability of the assigned ratings (e.g. Dempsey et al., 2009), but also make visible the learning objectives and provide useful feedback to identify strengths and weaknesses in a written product (Andrade et al., 2008). In this sense, rubrics may be employed for pedagogical purposes. In recent decades, much research has been conducted to analyze the potential of rubrics to promote students' learning processes (Wiliam, 2011).

Within the specific field of writing, rubrics may be employed to evaluate the quality of texts (e.g. Rezaei & Lovorn, 2010), but also to provide detailed feedback contributing to learning this process and to developing self-regulation skills related to writing (Saddler & Andrade, 2004). These aids used both for didactic and assessment purposes, are known as instructional rubrics (Andrade, 2001).

To date, few studies have been conducted to analyze the role of instructional rubrics in writing tasks. In the context of secondary education, Andrade (2001) conducted a study to analyze the impact of providing instructional rubrics on the quality of three different types of text (a persuasive essay, an autobiographical text and a fiction story). The results showed that students in the intervention group scored higher ratings than the control group, but only in the second task consisting of an autobiographical text. In another study, Sundeen (2014) examined the writing in persuasive essays by secondary school students, in three different intervention conditions: when given a rubric and explicit instruction on the criteria featured in the instrument, when provided with the rubric but without instruction, or when they were evaluated by simply applying the rubric. The results indicated that having access to the rubric (whether through receiving explicit instruction or being given a copy of such instruction) raises the quality of writing. However, no statistically significant differences were found between these two intervention conditions.

In the university context, Covill (2012) examined the effects of instructional rubrics when students wrote an argumentative essay after reading a text. Three intervention conditions were created: a first condition including an extensive rubric, a second condition with a reduced version of the rubric and a third condition in which students were asked to evaluate the strengths and weaknesses of their texts without the rubric. Students were instructed in the application of the rubric to evaluate a sample essay, and subsequently, applied it as appropriate to their experimental condition to evaluate their own outcomes. The results did not reveal any differences in the quality of written texts produced by students who received an instructional rubric (in its full or reduced version), or by the students who did not receive a rubric.

In a recent intervention study by MacArthur et al. (2022) to improve college writing, including the writing of argumentative essays, rubrics have been used to promote self-evaluation in the revision phase of writing, as part of a broader instruction aimed at strategy instruction with self-regulation. The rubrics contained genre-based evaluation criteria and were applied to model essays and peer review. The study found a large positive effect on the quality of student writing on an argumentative essay written as a final examination. However, because the evaluation rubric is only one instructional element within a more comprehensive curriculum, it is not possible to examine its separate contribution to the observed improvement.

To sum up, the results of earlier research in this field do not allow us to draw firm conclusions regarding the experimental conditions that modulate the impact of this type of instructional aids. Similarly, most of these studies have been conducted in the context of persuasive writing. No previous studies are known in which the impact of rubrics is studied, accompanied or otherwise by a process of explicit instruction, on learning to write argumentative syntheses from source texts containing conflicting information.

The present study

In this light, we propose going a step further to investigate the effectiveness of a new aid, likewise designed to assist in the process of writing argumentative syntheses: a rubric describing the integration strategies and quality criteria for texts written on the basis of these strategies. The specific aims of the current study were the following:

The first aim was to test the effectiveness of the rubric in comparison with the already tested guide, as aids in the process of writing an argumentative synthesis, both in the selection of arguments from the sources and in their integration. Given the lack of empirical evidence for the different effectiveness of these two types of aids, this aim is of a merely exploratory nature.

Since earlier research points to the greater effectiveness of such aids when explicit instruction is given in how to apply them, the second aim was to compare the effectiveness of the two aids under two different instruction conditions: with explicit instruction on the strategies to follow, and without such explicit instruction. We proposed, therefore, that both aids would lead to better quality writing, regarding both the selection and integration of arguments, especially when explicit instruction was given in their use.

Complementarily, the third aim was to analyze the effectiveness of the aids offered during the intervention, both in the two practice sessions performed with the support of aids, the guide vs. the rubric, and in the last session, in which the aids were withdrawn. Despite having only our own previous results on the use of the guide, we expected that after practicing the use of both aids students would produce better quality writing, and that this improvement would be extended to their final texts written without the support of aids.

Method

Participants

We carried out an experimental study with 120 second-year psychology students (17.5 % male and 82.5 % female), distributed randomly into four intervention programs. The age of the students ranged from 19 to 21 years with a mean of years 19.3 (Sd=0.49). All participants were native speakers of Spanish. Students volunteered to participate in this research, which was part of a writing workshop and for which they would receive some academic credit.

Instruments and materials

Texts for writing synthesis tasks

In this study we used four pairs of argumentative source texts, one for each session in the program (adapted from Granado-Peinado et al., 2023). Each pair of texts argued for different positions on several hotly debated topics in education (e.g., the initial training of teachers). All texts had a similar argumentative structure, with an equivalent number of arguments and counterarguments (between 8 and 9), a length ranging from 609 to 867 words, and a good readability index between 44.5 - 55.5, a range which is suitable for high school and university students (Szigriszt Pazos, 2001).

Intervention programs: components

We created four intervention programs, based on the combination of two instructional components: Type of instruction (EI - Explicit instruction vs. – WEI Without Explicit Instruction) and Type of aids to the process of writing an argumentative synthesis (Guide -G- vs. Rubric -R-). The third instructional component was common to all the programs and consisted of two individual practice sessions (P) in which the students wrote a synthesis with the help of the tool provided (Guide vs. Rubric). The different instructional components are explained below.

Instructional components

Guide (G)

The guide, an interactive tool consisting of a text with procedural steps and a graphic organizer, was adapted from previous studies (Mateos et al., 2018). Firstly, it provides an explanation for students to understand how to use this aid in the context of the task. Participants are then invited to explore both perspectives shown in the sources. To this end, they are encouraged to draw up a table and given advice regarding how to list the arguments and counterarguments, rank them and establish relationships among them, marking argumentative links with arrows.

In addition, it included four blocks of questions to guide students toward employing integration strategies to contrast arguments from both positions (weighing and synthesizing) and reach an integrative conclusion, and to plan and organize, textualize ideas and revise the final text. Link to access to the Guide: https://osf.io/mfvn7/?view_only=211b9f947a6140dfb8d7ca78b44094fd

Rubric (R)

The rubric includes a description of the criteria used to evaluate the quality of a synthesis, depending on the integration strategies used. It was adapted from previous studies (Casado-Ledesma et al., 2021). Firstly, as in the Guide, the rubric includes an explanation of the meaning of the tool in the context of the task. Secondly, it exposes the different integration strategies to construct an argumentative synthesis with an integrative conclusion, in which both positions are taken into account, comparing them with non-integrative strategies. Thirdly, the rubric includes a table that analyses in detail the criteria liable to enhance integration. The left-hand column indicates the level of integration (from 0 to 10). The next column details the integration strategy underlying each level. The remaining columns indicate the characteristics of the associated product written at each level: type of synthesis, depending on the number and nature of integrations made, throughout the body of the text (column 3) and in the conclusion (column 4). Link to access to the Rubric: https://osf.io/mfvn7/?view_only=211b9f947a6140dfb8d7ca78b44094fd

Explicit Instruction in synthesis writing strategies (EI) was adapted from Mateos et al. (2018). The objective of EI was to teach students how to reflect on and analyze different perspectives on a topic to create a synthesis with an integrating conclusion in which both positions are taken into account. To do this, we instructed students in the use of a guide or a rubric while planning, writing, and revising their syntheses.

Regardless of the aids provided, explicit instruction was focused on teaching integration strategies that could help students to write an argumentative synthesis with an integrative conclusion; in particular, we focused on - integration strategies via weighing and synthesizing - The integration strategies versus non integration strategies were explained and illustrated with examples. Table 1 includes examples of different strategies, such as arguing for one position, refuting the opposite position, weighing and synthesizing.

Table 1 Strategies and examples to write argumentative syntheses
Full size table

Given the nature of the guide and the rubric used, explicit instruction presented some differences depending on the aid provided.

Explicit instruction with the help of a guide was aimed at unravelling the different phases of the synthesis writing process: 1) identify and analyze in depth the different positions expressed in the source texts, identifying arguments from each position on the topic; 2) contrast and integrate arguments from both positions; 3) develop an integrative conclusion that considers both positions; 4) organize and textualize ideas in the written text; and 6) revise the written text. During the explanation, the recursive nature of the processes involved was emphasized. Link to access to the Explicit Instruction with the Guide: https://osf.io/mfvn7/?view_only=211b9f947a6140dfb8d7ca78b44094fd

By contrast, explicit instruction with the rubric focused on: (a) unraveling the different strategies that students can use to explore both perspectives and write their syntheses, ranking them according to the level of integration (non-integrative strategies, integrative strategies -via refutation, weighing or synthesizing-), and (b) relating the strategies used to the quality of the text produced (taking into account the number and nature of integrations made in the body of the synthesis and the type of conclusion reached). Link to access to the Explicit Instruction with the Rubric: https://osf.io/mfvn7/?view_only=211b9f947a6140dfb8d7ca78b44094fd

In the condition Without Explicit Instruction, the students received a training focused on the nature of an argumentative text and its different types and prototypical characteristics: structure, discursive mechanisms, types of logical-semantic and structural connectors, types of arguments and common errors in argumentation (adapted from González-Lamas et al., 2016).

Individual practice (P)

All participants completed two practice sessions. In each of these, students wrote an argumentative synthesis using the aid provided according to the condition assigned.

Design and Procedure

An experimental inter/intra-subject factorial design 2 (Type of instruction) x 2 (Type of aid) x 4 (Time) was employed. The dependent variable was the quality of students' argumentative syntheses (based on two criteria: integration level and argument selection -coverage of arguments-).

To control possible effects associated with the specific content of the sources, the presentation order of the 4 pairs of texts was counterbalanced, generating 4 sequences for writing the pre-test practice 1, practice 2, and post-test syntheses. Consequently, in each of the intervention programs (EI + P-Guide; EI + P-Rubric; WEI + P-Guide and WEI + P-Rubric) 4 subgroups were generated, in each of which 4 syntheses were written using the same texts but in a different order.

The students were randomly assigned to an intervention program and subgroup. The intervention programs were implemented by 4 instructors, members of the research team. Each instructor was assigned a subgroup of students in each of the intervention programs. This avoided instructor/program alignment, since the four instructors implemented the four intervention programs, albeit applied to different subgroups of students. The sessions in each intervention program are summarized below (see Table 2).

Table 2 Sessions in each intervention program
Full size table

Because of the health conditions deriving from COVID-19, all the sessions were implemented online, through the Teams platform. In the first and fourth sessions students performed an argumentative synthesis task without support aids (pre-post-test). The second session had two parts, in the first part, students received instruction according to the program to which they were assigned (EI vs WEI). In the second part of the session, each student wrote a new synthesis using the specific aid provided (Guide or Rubric) (Practice 1). The instruction for the task was: "Next, you are going to read two texts that defend two positions on a controversial topic in education. After reading the texts in the indicated order, you must write a synthesis, giving reasoned arguments to support your conclusion. To write the synthesis you may avail of the Guide or Rubric provided”. In the third session the participants carried out another synthesis under analogous conditions (individual practice 2, with aids given). All the sessions were completed in 4 consecutive weeks, one day a week.

After completing the program and for ethical reasons, participants who did not receive EI were invited to take part in an additional session, to implement it and explain the aids used in this study.

Treatment fidelity

In order to ensure intervention implementation fidelity in all 4 groups of students, a script was prepared with the content to be covered in each intervention program. Researchers in charge of each session followed this script, thus ensuring intra-program implementation equivalence. Participants were not allowed to intervene by asking questions or providing/sharing information during the sessions, to ensure that the information transmitted by the researchers was the same in each intervention program.

Coding system

The quality of students’ argumentative syntheses was evaluated on two criteria: integration level and argument selection (operationalized through the coverage of arguments).

Integration level: we used a ten-point scale adapted from previous studies (Casado-Ledesma et al., 2021) representing the type and frequency of argumentative strategies in the texts. This scale was the same as that used to draft the instructional rubric, with some additional specifications that allowed us to better discriminate the required characteristics in the products.

Coverage of arguments: we counted the total number of arguments included in the synthesis, based on a list constructed from the source texts.

Inter-rater agreement for integration and coverage of arguments was estimated with the Intra-class Correlation Coefficient (ICC) in a sample of 43 participants (35.8 % of the total sample, 12 % of the syntheses). Following the typology of Koo and Li (2016), a Two-Way Mixed-Effect Model of absolute agreement for the mean of two raters was calculated. In addition, Cronbach’s Alpha coefficient for consistency is reported. For integration, an Alpha = .917 and an ICC = .918, 95 % CI [.849 – .956] were obtained (excellent reliability). For coverage of arguments, an Alpha = .886 and an ICC = .885, 95 % CI [.788 – .937] were obtained (good reliability).

Data Analysis

Mixed effect models were carried out to test substantive hypotheses (i.e. to test the type of instruction, type of aid, and their interaction across the four moments on integration and coverage of arguments dependent variables). Maximum Likelihood was the estimation method used for fitting mixed models. To test fixed effects, we used the Likelihood Ratio Test, AIC and BIC selection model indices. The equivalence of the groups was verified for the dependent variables at the pre-test stage, no significant differences were found between the experimental conditions. Despite achieving equivalence, the results were also examined by adjusting mixed models for the pre-test scores to ensure robustness in the findings.

Results

Descriptive analysis

Means, standard deviations and medians are presented in Tables 3 and 4. In general, there was an increase in the means for integration and coverage of arguments in the two practical sessions with respect to the pre-test moment. Post-test measurements revealed a slight decrease in integration, this pattern becoming less evident for coverage of arguments (where post-test means are the highest in some of the conditions).

Table 3 Means (standard deviations) and medians of integration across the four studied moments for each type of instruction and aid received
Full size table
Table 4 Means (standard deviations) and medians of coverage of arguments across the four studied moments for each type of instruction and aid received
Full size table

Mixed models

Integration

Table 5 shows different nested models that differ in their fixed effects (all mixed models share the same random-effect structure: heterogeneous compound symmetry covariance structure and residual level-1 variance).

Table 5 Nested mixed models for integration dependent variable with AIC, BIC and LRT model selection indices
Full size table

According to model selection indices (except for the BIC index which favours the model with time as the only-fixed effect in the model, perhaps because it over-penalizes complex models), the best model explaining integration level features Time and Instruction as main fixed effects and Time x Instruction interaction effect (model [3] in the table). This model includes two significant main effects: time (F(3,211)=46.800, p < .001) and the variable Type of Instruction (F(1,121)=5.993, p = .016), as well as a marginally significant interaction effect between time and type of instruction (F(3,211)=2.340, p = .074). Figure 1 helps to clarify these significant main effects and the marginal interaction effect. Integration means vary across the study and EI yields a significantly higher mean than WEI (0.686 points higher on average 95 % IC [0.115 – 1.256]). Multiple adjusted Bonferroni comparisons show that integration was higher in the second practice (even with respect to first practice, p = .013, and with respect to post-test moment, p = .037). The pre-test integration means were significantly lower than in the other three moments (p < .001). Simple effects for Time factor show significant differences between post-test and pre-test moments, both in EI (average difference of 2.42 points higher at the post-test than at the pre-test, se = 0.360, p < .001, d = 0.839, large effect size) and WEI (average difference of 1.554 higher in the post-test than in the pre-test, se = 0.403, p = .001, d = 0.492, medium effect size). In addition, to assess the removal of aid in the post-test, we found a marginally significant decrease in the WEI group from the second practice to the post-test (p = .092). However, there were non-significant differences when comparing second and post-test moments in EI group (p = .870). Finally, analyzing the simple effects for Type of instruction reveals that significant differences exist between the two types of instruction at two critical points. Firstly, during the initial practice, there is a notable improvement of 1.281 points favouring explicit instruction (se = .446, p = .005, d = 0.516, medium effect size). Importantly, this advantage underscores that the participants receiving explicit instruction not only perform better initially, but also demonstrate significantly greater integration once the supports are withdrawn. This is further evidenced in the post-test, where the difference of 0.915 points still favours explicit instruction (se = .458, p = .048, d = 0.360, medium-low effect), indicating sustained benefits of explicit instruction methods over those without explicit instruction. Non-significant differences were found between the two types of instruction in the pre-test and the second practice moments.

Fig. 1
figure 1

Means and error bars (95% CI) for integration. The horizontal axis shows Time factor and lines represents two Types of instruction

Full size image

Coverage of arguments

Table 6 shows competing nested models that differ in their fixed effects (all these models share the same random-effect structure (in this case homogenous compound symmetry covariance structure). As the incorporation of main effect terms is arbitrary, we have inserted the Aid factor before the Instruction factor in this table (for ease of interpretation).

Table 6 Nested mixed models for coverage of arguments dependent variable with AIC, BIC and LRT model selection indices
Full size table

In this case, model [2] is preferred (the smallest AIC, BIC and the only significant LRT). Figure 2 helps to understand this model. This model identifies two significant main effects: one for time (F(3,360) = 4.179, p = .006) and another for Type of Aid (F(1,120) = 8.610, p = .004) (non-significant interaction was found between time and Type of Aid (F(3,360) = 0.383, p = .765) . As illustrated in Fig. 2, the means in the pre-test indicate differences between samples that could potentially influence or confound these significant effects. After adjusting for pre-test scores, the revised model still indicates a significant main effect for Type of Aid (F(1,120) = 7.161, p = .008) and a highly significant effect of the pre-test scores (F(1,120) = 33.955, p < .001). Additionally, a marginally significant effect for time was observed (F(1,240) = 2.391, p = .094). The coverage of arguments with guide yielded a significantly higher mean than with rubric (0.907 points higher on average 95 % IC [.289 – .1.525]). With respect to the time factor, there were non-significant differences in the coverage of arguments between pre-test and first practice moments. In addition, significant differences arose between the pre-test and second practice (0.767 higher in second practice) and between the pre-test and post-test moments (0.617 higher in post-test moment). Simple effects for time factor evinced a significant effect with guide (p = .023), but not with rubric (p = .277, where the differences between any pairs of moments do not reach statistical significance). With the guide there was a marginally significant difference between post-test and pre-test moments (average difference of 0.857 points higher at the post-test than at the pre-test, se = 0.347, p = .084, d = 0.295, low-medium effect size) and a significant difference between second practice and pre-test moments (0.929 points higher at the second practice than at the pre-test, se = 0.347, p = .050, d = 0.317, low-medium effect size). In addition, simple effects for Type of aid, in the pre-test moment and first practice, showed non-significant differences between the two types of aid (p = 0.100 and p = .070 respectively), but significant differences between types of aid in second practice (a difference of 1.004 points in favour of guide, se = .425, p = .019, d = 0.450, medium effect size) and in the post-test moment (a difference of 1.152 also favouring guide, se = .425, p = .007, d = 0.495, medium effect size). When the simple effects are examined and adjusted for pre-test scores, the results are replicated (statistical significance was found in the second practice and in the post-test), although the corrected differences between the guide and rubric were 0.781 and 0.929 in favour of the guide.

Fig. 2
figure 2

Means and error bars (95% CI) for coverage of arguments. The horizontal axis shows Time factor and lines represents two Types of aid

Full size image

Discussion

This study aims, firstly, to compare the effectiveness of two aids -guide and rubric- to improve the quality of the syntheses written by undergraduates from texts that defend opposing views on a topic. These aids are designed to mediate the process of writing argumentative syntheses, both in the selection of arguments from the source texts and in their integration. Secondly, to ascertain the extent to which explicit instruction in synthesis writing strategies enhances the effects of such mediation to improve the quality of the argumentative syntheses written by students And lastly, to study in greater depth the instructional conditions that favour the appropriation of the aids provided during the intervention, comparing the quality of the synthesis in the two criteria under study, selection and integration, during the two individual practice sessions supported by the written tools (rubric/guide), and when the aids were removed.

Regarding the first aim, our results suggest that both the guide and the rubric were useful aids to improving the integration strategies applied by students when writing their argumentative syntheses. Prior to the intervention, half of the students wrote their synthesis adopting a stance, using what Nussbaum (2008) called ‘one-sided reasoning’, including arguments from a single source to argue in favour of their position (a Median score of 2). They also included arguments from the opposite position but only for the purpose of discrediting it (using the refutation strategy) (a mean score of 3). By contrast, after receiving the guide or rubric, students showed greater integration in their syntheses, incorporating arguments from both perspectives, and considering throughout their synthesis the benefits, disadvantages, and conditions associated with each viewpoint. In general, the students improved from a mean score of 3 (refutation) to 5 or 6 (weighing).

However, regarding the selection process, only the groups provided with the guide significantly increased the number of arguments and counterarguments included in their syntheses. The rubric had no effect on improving this process. In this case, the practice session supported by the guide proved essential to improvement, independently of whether the aid was preceded or not by explicit instruction. The guide prompted the students to complete the graphic organizer, identifying the arguments and counterarguments for both positions, and link them with arrows. This resulted in a greater number of arguments selected. By contrast, with the rubric, assistance with the selection process remained subordinated to the process of integration. The rubric focused more on the integration process. The different emphases of the guide and rubric in the selection process likely explain the contrasting results.

As for the second aim, and in accordance with our hypotheses, the students receiving explicit instruction in synthesis writing strategies, independently of the aid provided during the practice sessions, wrote syntheses with a higher degree of integration than those by students who likewise availed of the aids during the practice sessions but did not receive this explicit instruction. In line with other research, students who used the aids after receiving explicit instruction were able to overcome the “my-side bias” (Wolfe et al., 2009) and apply in their final text synthesis writing strategies involving balanced reasoning, such as “weighing” and “synthesizing” (Nussbaum & Schraw, 2007), both in the body of the text and their conclusions after weighing the pros and cons of both perspectives. They were able to produce syntheses with intermediate levels of integration, generating integrative conclusions (Mean 5.47, Median: 5)

However, the students who used the aids without explicit instruction still made some progress, analyzing and integrating certain arguments and counterarguments (through weighing or synthesis) throughout their texts, but fell short in generating integrative conclusions. Students wrote post-test syntheses with a basic level of integration bringing the groups' average below 5 (Mean 4,45, Median: 4).

These findings align with previous research indicating that the effectiveness of guides is enhanced when synthesis writing strategies are explicitly taught. These results are consistent with those obtained (a) in face-to-face teaching contexts (Granado-Peinado et al., 2023; Mateos et al., 2018), and (b) in virtual contexts (Luna et al., 2023).

Regarding the impact of an instructive rubric on the writing of argumentative texts, our results are coherent and enlarge upon other researchers' findings. The use of an instructive rubric enabled students to assimilate certain integration strategies, thus improving the degree of integration in their final syntheses. As in the study by Sundeen (2014), the rubric designed and provided to students contextualizes and explains the criteria and levels of execution according to the strategies deployed. Communicating the meaning of the rubric in the context of the task and explaining the criteria and levels of execution may contribute to enhancing the quality of the ensuing texts.

Certain circumstances may also affect the impact of aids (guide or rubric) on synthesis quality and help to explain the diversity of results obtained in the field. On the one hand, task complexity -a reflexive essay as opposed to a persuasive essay- (Nussbaum, 2008), and on the other hand, students' initial synthesis writing strategy skills (Granado-Peinado et al., 2023), as well as their ability to use strategies for tackling the demands associated with planning and revising their essays (cf. Ferretti & Fan, 2016).

Our study suggests that aids such as the guide and rubric are suitable for students when writing high cognitive complexity essays, such as argumentative syntheses with a reflexive and integrative aim. Explicit instruction in synthesis writing strategies, supported by written mediators, is crucial in such cases. However, for less complex processes like argument selection, students already benefit from the specific mediators provided during practice sessions, such as graphic organizers and guide questions.

In terms of the ownership and internalization of the aids, our third aim, the results regarding the effect of instructional components on the integration process showed that in the first practice session, with the support of the aids provided (guide or rubric), the group receiving explicit instruction benefited more from the aids than the group receiving traditional instruction. However, in the second practice session, all groups showed similar levels of integration, regardless of prior instruction. Nevertheless, the learning differed based on the instruction received. Students who received explicit instruction maintained their integration skills even after the aids were withdrawn, while those without explicit instruction experienced a decline in integration. Explicit instruction was key to appropriation and adoption of complex synthesis integration strategies, after the aids were withdrawn (post-test). These results contrast with those obtained in studies by Nussbaum (2008) in which progress in the use of more integrative argumentation strategies (weighing and synthesis) was observed when students were given a pre-writing tool and participated in a group discussion, but disappeared when the graphic tool and group discussions were removed. In our case, and although the practice session with the aids was individual rather than collaborative (Mateos et al., 2018; Mateos et al., 2020), the removal of the aids did not cause the integration strategies used in the practice session to disappear, especially in the group that received explicit instruction in which case the learning was greater.

As for the selection process our results indicate that, after adjusting pre-test selection scores in the mixed model, an improvement occurred in the conditions in which the practice was conducted with the support of the guide, regardless of the instruction received. By contrast, practice with a rubric (with/without instruction) did not assist with this process.

Our results are coherent with those obtained in other studies investigating the direct or indirect effect -through collaborative practice sessions with the support of a guide- of explicit instruction on two argumentative synthesis quality indicators (selection and integration) (Granado-Peinado et al., 2023; Mateos et al., 2020). In these studies, researchers found the direct impact of explicit instruction on the degree of integration in intermediate syntheses written with the support of the guide, and in final syntheses written without aids. In the case of argument selection, previous studies show that intermediate practice sessions with the support of a guide mediate the full impact of explicit instruction on the number of arguments and counterarguments selected by students from the sources and included in their final syntheses (Granado-Peinado et al., 2023). In our study we also found that practicing with the support of a guide was the key component allowing students to increase the number of arguments and counterarguments selected from both sources.

Limitations and implications for future studies

The findings revealed that two practice sessions supported by a guide or rubric, together with explicit instruction, helped students who initially struggled with integrative strategies (i.e. students who argued in favour of one position or refuting the opposite). After the intervention, these students were able to consider multiple perspectives and produce syntheses with intermediate levels of integration, weighing and integrating arguments in the body of their syntheses. However, the instructional aids did not lead to most students achieving a high degree of integration (level 8 to 10). As mentioned above, the integration process -especially via synthesis- is a highly complex process. We know that to provide support to students it is necessary to adjust the aids to their skills and re-adjust them according to the progress made. Once the students have understood what is demanded in the task and have also gained the capacity to de-centre and approach the task in a reflective and epistemic mindset, it becomes necessary to adjust the type of mediators for these to be effective.

Therefore, further research is required -cross-sectional (according to the students' initial skills) and longitudinal studies (according to their evolution)- to analyze what aids, and which adjustments are needed, so that the aids contribute to consolidating and improving certain processes, such as a deeper exploration of the different perspectives on a topic and enhancing their strategies for generating integrative conclusions.

Similarly, knowing the difficulties and strengths of the aids employed may contribute to incorporating improvements. In this study we did not analyze how students used the aids. In this sense, the conduct of in-depth interviews or discussion groups would enable a better understanding of the usefulness of the aids provided, what further resources they would have needed to make progress in synthesis writing and how the aids facilitated the regulation of the synthesis writing processes.

In this regard, some studies have reported that the textualization process is particularly complex and, therefore, diversifying and enlarging the aids to enable students to improve their integration of arguments and counterarguments throughout their texts and, most especially, in their conclusions may be a necessity (cf. Luna et al., 2023). In this respect, writing integrative conclusions appears to be a particularly complex process for students. Future studies should also focus on this aspect.

Conclusions and Educational implications

The poor level of integration in our students' first syntheses suggests that teachers should be aware that the inclination of most students' when dealing with controversial sources of information is to adopt one of the sides, unless they have been taught to discriminate between writing texts with a persuasive or reflective purpose. This approach to the task is still a distance away from critical thinking, which is essential in higher education. Hence, the importance of this instruction is evident.

The results also clarify which type of instruction may be effective, making distinctions between the various processes involved. To select arguments and counterarguments, it is sufficient to practice with the support of the guide, probably with a focus on using the table intended for identifying and relating arguments. It might be appropriate, therefore, to firstly focus instruction on this process, and follow on with explicit instruction in the use of the guide, with emphasis on integration, writing and revision strategies. Teachers should understand that practice, even with the support of the guide or rubric, is not enough to write integrative syntheses. Explicit instruction significantly improves the quality of such texts, and more importantly, allows the consolidation of learning after the aids are withdrawn.

An instructive rubric which not only outlines levels of attainment but also unravels the most appropriate integration strategies, is a very useful tool for teaching argumentative synthesis writing. The rubric allows students to evaluate their results and effectively plan and oversee the writing process. However, it should be preceded by the use of a graphic organizer which could improve the selection and comparison of arguments.

The instructional aids designed, both the guide and the rubric, are a valuable contribution of this work. In future research, it would be interesting to replicate this intervention with secondary and upper elementary students. Casado-Ledesma et al. (2021) demonstrated the usefulness of a guide designed for third-year secondary students. It would also be necessary to check the functioning of the rubric adapted to these ages. Critical reading, writing, and thinking are relevant goals across secondary education. It is important to note, however, that these epistemic writing skills must continue to be developed at the university level as an essential component of academic training.