Abstract
Self-regulation has a positive impact on learning and academic achievement, but due to its nature, it is difficult to assess it in a valid and reliable manner. This study aims to explore the validity of three self-regulation assessment methods in text comprehension tasks (questionnaire, think-aloud and traces) as well as to identify the variables associated with reading performance. For this purpose, individual sessions were conducted with 96 sixth-grade primary education students, who were asked to read a text and complete two reading comprehension tasks working in a virtual environment. Task traces were recorded, think-aloud was recorded and coded, and participants answered a self-report questionnaire about the strategies used. A limited relationship was found between self-regulation measured by means of the questionnaire, think-aloud and traces, and the relationship between think-aloud and traces was moderate. Regarding the factors related to text comprehension task performance, think-aloud and task time measurements were found to be better predictors of performance than the questionnaire.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Self-regulated learning has a positive impact on learning and academic achievement (Dent & Koenka, 2016; Moos & Ringdal, 2012). Self-regulated learning is not a cognitive aptitude or an academic competency; instead, it represents the self-directed process through which learners translate their cognitive abilities into academic skills (Zimmerman, 2002). These self-regulated processes involve systematically governing thoughts, feelings and actions to achieve an academic goal (Stoeger & Zeidner, 2019).
The relationship between self-regulated learning and text comprehension has been thoroughly studied (Dignath & Büttner, 2008; Okkinga et al., 2018). Text comprehension is a key competency for attaining lifelong learning goals. In particular, in the last years of primary education, the need to read expository texts to learn specific content increases (Rogiers et al., 2019). This competency involves putting into play a series of cognitive, metacognitive, affective and behavioural processes and regulating these multiple processes to reach the goal of comprehension. Self-regulatory processes have important effects on text comprehension (Steiner et al., 2020; Wu et al., 2021). One of the key differences between good and poor readers lies in the effective use of self-regulatory processes to control the reading process (Heirweg et al., 2019; Wang, 2016). The need to implement self-regulatory processes for text comprehension increases in tasks that involve reading to learn (Hu & Gao, 2017).
Self-regulatory processes in reading comprehension
Many of the theories about self-regulated learning view the structure of self-regulatory processes in terms of a set of cyclical phases: forethought, performance and self-reflection. Each of these three phases consists of different processes that a student goes through when performing a learning task (Zimmerman, 2011). This cyclical nature has been affirmed in reading tasks by Heirweg et al. (2020), who observed that students go back to a previous strategy or phase when understanding a text.
In the forethought phase, students go through two main processes: task analysis and self-motivation. To analyse a task, students think about what is required, identify the goal and assess whether the goal is deep comprehension. Additionally, they think about what they already know about the topic or about similar tasks they have performed (Butler & Schnellert, 2015). In this phase, as outlined by Zimmerman (2011), students’ self-efficacy beliefs regarding their personal capacity to learn and their expectations about the personal consequences of learning become significant factors (Zimmerman, 2002).
During the performance phase, processes can be categorized into two primary classes: self-control and self-observation. In self-observation, one of the most relevant subprocesses for task achievement is monitoring, which involves an online cognitive process for evaluating performance (Panadero & Alonso-Tapia, 2014). It has been observed that it is especially used by high-performing students (Denton et al., 2015; Steiner et al., 2020). It also seems to play an important role when reading online (Burin et al., 2020).
Performing a text comprehension task entails a degree of complexity that demands the activation of self-control processes, including self-instruction, such as giving oneself instructions, asking questions or providing explanations (Eme et al., 2006; Zimmerman, 2011). Self-instructions have been identified as having positive effects on both motivation and performance (Oliver et al., 2008). Moreover, their usage tends to increase as the task becomes more demanding (de Guerrero, 2018).
Regarding the last phase established by Zimmerman (2011), self-reflection, there are two major classes of processes: self-judgment and self-reaction. The primary form of self-judgment, self-evaluation, involves comparing one’s observed performances against a predetermined standard (Zimmerman, 2002). Self-evaluation about both the outcome and the process is a subprocess that has proven to be efficient for improving reading comprehension (RC) (Saat & Özen, 2022).
In text comprehension tasks, it has been found that students mainly use performance phase processes and make very little use of forethought and self-reflection processes (Heirweg et al., 2019).
Assessing self-regulatory processes
There are differences in the use of self-regulatory processes that make good or poor readers. Wang (2016) conducted a think-aloud (TA) study with ten students who worked on reading tasks in pairs. This study found differences in the processes used by the worst-performing and the best-performing pair. In general, the best-performing pair displayed a better use of reading strategies, sound linguistic knowledge, use of background knowledge on the topic of the text, monitoring of comprehension and integration of the text meaning. On the other hand, the worst-performing pair stood out for the number of times they resorted to seeking help. On the other hand, Heirweg et al. (2020) found that the best-performing students are more strategic and flexible in their learning, manage to direct the task towards a goal, use cognitive strategies and self-reflect to regulate their process.
These studies that describe the differences between good and poor readers are of great importance to inform intervention programs, since self-regulation and strategic reading are processes that can be taught explicitly and systematically (Dignath & Veenman, 2020). It is then necessary to evaluate self-regulated processes in reading tasks to inform and assess intervention programs. The evaluation of self-regulated processes in schoolchildren is a challenge for research in this area due to the difficulties that exist in evaluating it in a valid and reliable manner, given that it deals with internal, not openly explicit processes (Panadero et al., 2016).
The most widely used self-regulation assessment method is the questionnaire. This offline measurement has the disadvantage of the students’ lack of precision when describing and reporting their processes (Panadero et al., 2016) and the difficulty of capturing the changes that can occur in self-regulated processes during a learning task (Rovers et al., 2019). When used with primary school students, there may be difficulties in generalizing what their typical way of approaching tasks is, and there is usually a positive bias in their answers, many of them being very optimistic about their performance (Perry et al., 2010). Van Ammel et al. (2021) cite having used questionnaires as the only tool as a limitation in their study about the influence of motivation and the use of strategies on text comprehension performance. They propose to extend assessment using online measurements such as think-aloud, traces and eye-tracking.
TA is an online method for the evaluation of self-regulated processes. In this type of assessment, self-regulated processes are conceptualized as an event closely related to the characteristics of the task (Panadero et al., 2016). It consists of asking participants to think aloud while executing a task and identifying the presence of self-regulated processes from their verbalizations. This method has been used in numerous studies to assess self-regulated processes in text comprehension tasks (de Milliano et al., 2016; Denton et al., 2015; Heirweg et al., 2019; Máñez et al., 2022; Rogiers et al., 2020). These studies highlight TA as a useful tool for evaluating self-regulation in this specific domain, since it allows access not only to students’ overt strategies, such as rereading or highlighting, but also to covert strategies, such as monitoring or strategic planning (Rogiers et al., 2020). Unlike questionnaires, it does not depend on students’ memory and is less vulnerable to distractions (Heirweg et al., 2019). These protocols have proven to be efficient for understanding the differences between high- and low-achieving students, as well as for evaluating self-regulated reading processes before and after a pedagogical intervention (Hu & Gao, 2017).
This methodology may also have its limitations and problems. First, reporting TA while performing a task can have an effect on thoughts. In their review of studies on the use of self-regulation assessment through TA analysis, Hu and Gao (2017) found that this alteration may be mediated by variables such as the age of the participants—it may be higher in children—and by the type of task: this interference can be amplified in reading tasks. Meanwhile, Máñez et al. (2022) found that students who applied the TA method when answering questions about a text spent more time searching for relevant information. Nevertheless, students can be prepared for TA through examples and practice sessions, which can help minimize these difficulties (Hu & Gao, 2017). Finally, this method requires time and resources because it involves working individually with each participant.
In order to access what students do not verbalize, another form of online evaluation is the analysis of traces. This involves recording the student’s behaviour when faced with the task and can be used to infer aspects related to self-regulated processes. Traces can be evaluated very accurately using software (Perry et al., 2010). Numerous studies have used this assessment method (Ferrer et al., 2017; Minguela et al., 2015; Rogiers et al., 2020; Van Der Graaf et al., 2021; Van Halem et al., 2020). Rogiers et al. (2020) carried out a study with 56 high school students comparing self-regulated processes evaluated by means of a questionnaire, TA and traces, which consisted of analysing the notes and marks that the students left on their worksheets. These authors found low to moderate correlations between the processes reported in the questionnaire and those coded in TA data. The same occurred for the correlations between self-report and traces. No significant correlations were found between the self-report of metacognitive processes and their verbalization. In another study with 79 students, Ferrer et al. (2017) automatically recorded students’ actions when reading a text and answering questions on a digital platform: timing, referring back to the text etc. They found that students refer back to the text more frequently when answering open-ended questions than when answering multiple-choice questions, and that going back to the text helped students perform better in the open-ended question task. On the other hand, in a study with 55 students, Minguela et al. (2015) recorded indicators of the reading process such as what parts of the text the students read when they referred back to the text, and how much time they took.
This study aims to establish relationships among three methods of self-regulation assessment (questionnaire, TA and traces). It seeks to explore the validity of these three methods in assessing self-regulated processes during reading comprehension tasks, examining their relations with other variables (American Educational Research Association, 2011). Specifically, the study examines the relationships between the methods to determine convergent validity and their connections to text comprehension performance to establish predictive validity. Additionally, the study aims to identify the indicators that most effectively explain text comprehension performance within each method. It aspires to contribute to the field of research in self-regulation, addressing its assessment, which has been noted as one of its great challenges, by responding to the need of triangulation of online and offline data, as noted in the literature (Beek et al., 2019; Harding et al., 2019; van de Pol et al., 2019). Moreover, there is an endeavour to ascertain the indicators in each assessment method that contribute to explaining task performance, an area that has not been thoroughly investigated in the existing literature.
Method
Objectives
-
Describe the self-regulatory processes used in text comprehension tasks by students in the last year of primary education.
-
Relate the three self-regulation assessment methods in reading comprehension tasks.
-
Determine the self-regulation assessment methods that best predict performance in a reading task.
-
Identify student profiles according to the use of self-regulated processes.
Participants
The study included 96 schoolchildren in the last year of primary education, from three purposefully selected schools in Uruguay. All are urban schools of medium socioeconomic level. Fifty-three percent of the students were male and 47% female; 97% were 11 years old, and 3% were 12 years old. The average age of the students was 11 years and 3 months, and Spanish was the native language for all participants.
Instruments
A digital platform, specifically designed for this study, was used to assess text comprehension. It contains a text, six multiple-choice questions and an open task that consists of making a summary. The aim was for the text and the tasks to be similar to those that are usually used in the classroom. The text is a 484-word expository text that addresses a sixth-year curricular topic: electrical energy. Teachers were asked about the students’ prior knowledge regarding the text’s topic. The teachers informed that they were just beginning to address the topic in class at that time. The text was reviewed by three RC evaluation experts to ensure it was challenging, yet understandable for students. A scale from 0 to 15 was used to score the text summaries made by the students, according to a rubric prepared for this purpose based on the criteria of León et al. (2015), which evaluates content, coherence and synthesis (see Table 1).
The same digital platform was used to assess self-regulation by recording traces. The following traces were automatically recorded: use of the word meaning aids, text queries, changes in answers to comprehension questions and task performance time (Ferrer et al., 2017).
An abbreviated version of the “Text-Learning Strategies Inventory” questionnaire (Merchie et al., 2014), developed and tested with fifth- and sixth-grade students, was used. The original questionnaire has 37 items divided into nine subscales. It is designed to be answered after reading an expository text using a 1 to 4 Likert scale (strongly disagree, slightly disagree, slightly agree, agree). To obtain the reduced version, 13 items were selected, specifically associated with forethought, monitoring and evaluation, which correspond to the Zimmerman (2011) cyclical model. The translation was carried out after consulting experts to validate it. This version was applied to 284 students in the last year of primary school in the metropolitan area of Montevideo. A confirmatory factorial analysis was carried out using AMOS 26 and the best model was found to be that of eight items in a single general self-regulation factor; goodness-of-fit indicators were as follows: (χ2 = 34.31, p < 0.001, CFI = 0.93, IFI = 0.93, TLI = 0.89, RMSEA = 0.05). Cronbach’s alpha coefficient of the questionnaire was 0.64.
To assess the self-efficacy of students in text comprehension tasks, the MSLQ self-efficacy scale adapted to the Uruguayan population was used (Curione et al., 2017). Items were answered on a 5-point Likert scale (never-rarely-sometimes-often-always). The six items from the validated version of the MSLQ self-efficacy scale for Uruguayan university students were adapted to text comprehension tasks (e.g. “my reading skills are excellent”). Sixth-grade teachers were consulted about the relevance of using this scale for their students. The scale yielded a Cronbach’s alpha coefficient of 0.77.
To assess the reading efficacy of the students, the TECLE test was used (Cuadro et al., 2009). This test assesses reading accuracy and speed (Balbi et al., 2009), has been standardized for the Uruguayan population and is widely used in reading studies in Spanish (Ochoa et al., 2019; Tabullo et al., 2022).
Procedure
The evaluation platform was designed with the advice of experts who reviewed the clarity of instructions, the degree of difficulty of the tasks and the text, the relevance for sixth-grade students and the use of the platform. Based on this review, modifications were made to the instructions, the vocabulary and the design. This new version was tested on 76 students. With the information provided by this pilot test, the records, notifications and the interaction with the platform were improved and one task was eliminated. Finally, it was tested on four other students and adjustments were made to the digital design.
The TECLE test and the self-efficacy scale were administered collectively by the main researcher. In both cases, instructions were explained, using an item as an example. Subsequently, individual sessions were held with the participants by three evaluators previously trained. These sessions lasted an average of 30 min and 50 s, ranging from 14 to 60 min. Each individual session started with TA modelling, following guidelines from previous research (Hu & Gao, 2017). The modelling was conducted through a video where the main researcher thought aloud while solving a tangram puzzle. A practice task lasting approximately 10 min was then carried out, where students were asked to think aloud while solving a virtual tangram puzzle. During this practice activity, the evaluators gave feedback to the participants. In this way, students practiced thinking aloud in tasks other than text comprehension, as recommended by Hu and Gao (2017) in their review of TA studies.
The students then worked on the digital platform. During task performance, students were observed and recorded via Zoom from another part of the room. At all times, efforts were made not to interfere with their spontaneous cognitive processes. Finally, immediately after completing the tasks, the students responded to the self-regulation questionnaire virtually, after an explanation and an example by one of the evaluators.
Ethical aspects
All the students’ families submitted their free informed consent. The students were informed about the study’s objectives and procedures. A detailed explanation was provided regarding the time commitment and tasks involved, and they were assured that they could withdraw at any point. The students agreed to participate. The study was approved by the ethics committee of the Autonomous University of Madrid (CEI-110-2165).
Data analysis
The digital tools PinPoint and oTranscribe were used for transcription. The transcript included gestures, expressions and actions.
A deductive analysis of the contents was carried out, starting from predefined categories that consider the self-regulated processes contemplated in the Zimmerman (2011) model (see Table 2). This cyclical model of self-regulation of learning classifies self-regulated processes into three phases: forethought, performance and self-reflection. A coding protocol based on this theoretical model was designed for this purpose. The verbalizations were codified using the QDA Miner software. Units of meaning ranging from a statement to a paragraph were identified. The transcription included 48 h and 59 min of video. In Table 2, the category system and examples of the transcriptions can be seen.
Ten percent of the protocols were double-coded by two independent and trained coders. Agreements were reached on the categories that were confusing. The protocols were recoded resulting in a reliability of 0.76, according to Cohen’s kappa. Each evaluator then coded the rest of the participants.
The frequencies for each category were obtained and the percentages of the total verbalized strategies were calculated.
The text summaries were scored by the principal investigator and two evaluators using a rubric based on León et al. (2015). Inter-rater reliability was evaluated using the Pearson correlation coefficient, which was initially 0.77. The summary score was taken as the best performance indicator in text comprehension, since the task scores showed good variability and significantly correlated with reading efficacy in the TECLE test. The correlations exhibited values higher than 0.29.
An exploratory factor analysis was performed to determine which TA and trace variables were the most discriminant. The Kaiser-Meyer-Olkin test value was below 0.50 (specifically, a KMO value of 0.06 was obtained); consequently, the procedure was not deemed appropriate. Therefore, the correlations were calculated between TA variables, traces and the questionnaire and between these and task achievement, measured as the summary scores.
Regression analyses were performed using the forced-choice method to study the relationships between the three self-regulation assessment measures. First, the self-regulation questionnaire was used as the dependent variable, and the TA categories were selected in the previous phase as the independent variable. Second, task traces (TT) were used as an independent variable: summary performance time and text queries. Subsequently, a regression analysis was performed with one of the traces (performance time) as the dependent variable and the TA categories as the independent variable.
In order to determine the variables from the three assessment methods that best explain performance in the task, forced-choice regression analyses were conducted, with the summary score as the dependent variable. First, the questionnaire was used as an independent variable; in a subsequent model the measurement of traces (performance time) was added; and in the last model, TA measurements were also added.
K-means cluster analyses were performed considering the self-regulation strategies measured by the TA. In order to relate the groups produced by the cluster analysis with the task performance time and the summary score, a mean comparison t-test was performed.
Results
Table 3 shows the descriptive statistics for all the TA categories, TT, the RC performance scores and the self-regulation and self-efficacy questionnaires.
Students reported a high use of self-regulation competencies in the questionnaire. Achievement in the RC task was average. Think-aloud data analysis showed that the strategies that the students used the most correspond to the performance phase in 84% of their verbalizations, followed by the forethought phase (11%) and finally the self-reflection phase (5%). The most used strategy was monitoring (36%), and the least used were those related to motivational aspects.
In order to estimate the convergent validity, Pearson correlations were calculated to correlate the different self-regulation measurements (TA, traces, questionnaire). The self-regulation questionnaire showed moderate-to-low statistical correlation with the sum of the TA strategies, with a value of r(89) = 0.24, p = 0.02. No correlation was found with any of the specific strategies or with the group of strategies of each phase measured using TA. The questionnaire measures self-regulation globally, which is consistent with the fact that it correlates with the total TA measure.
When relating the questionnaire with TT, significant moderate-to-low correlations were found both with the time that the students took to complete the summary, with a value of r(89) = 0.33, p = 0.002, and with the number of times they referred back to the text throughout the task, r(89) = 0.25, p = 0.004.
The correlations between total TA measurement and traces were moderate, both with task performance time r(89) = 0.45, p < 0.001, and with text queries r(89) = 0.31, p = 0.002. Moderate correlations were also found for help-seeking strategies, r(89) = 0.27, p = 0.01, with task performance time, and for strategic planning strategies with text queries, r(89) = 0.30, p = 0.003.
As another way of analysing the convergence between the three measures, a multiple regression analysis was calculated using the enter method, with the self-regulation questionnaire score as the dependent variable and the TA categories as the independent variable. The regression equation was not statistically significant, F(5.84) = 1.34, p = 0.26. The R2 value was 0.07. A new regression model was performed with the same dependent variable and TT as independent variables (summary execution time and text queries). The two independent variables showed collinearity problems, so the text queries variable, which had a lower correlation, was excluded. The regression equation was statistically significant. F(2,84) = 10.67, p = 0.002. The value of the R2 was 0.12, and the effect size according to Cohen was medium (β = 0.33). In short, there is only a predictive relationship when the performance time and the self-regulation questionnaire are taken into consideration.
A third multiple regression model was carried out, taking the most discriminant trace measure (time) as the dependent variable and the TA measures as independent variable. The best model obtained is the one that includes the measures of strategic planning (TA), specific task strategies (TA), self-instruction (TA), help-seeking (TA) and control (TA), F(5.84) = 4.72, p = 0.001. The R2 value was 0.22 and the effect sizes were low according to Cohen (β = 0.23).
In order to determine the predictive validity indicators of self-regulation processes that are related to task performance, the correlations of the variables of the three self-regulation measures (traces, TA and questionnaire) with the summary score was calculated. Correlations with self-efficacy were added as another factor that can be linked to achievement in the RC task. Table 4 shows the variables that presented significant correlations. No correlations of the other traces (word meaning aids and response changes) were found with any of the variables studied.
As can be seen in Table 4, there was no correlation between performance in the RC task and the self-efficacy perceived by the students.
Regarding TA categories, correlations with RC performance score are shown in Table 5.
The summary score significantly and moderately correlated with the total strategies, self-instructions and task-specific strategies and also correlated in a significant, albeit low, manner with strategic planning and negatively with help-seeking.
Three linear regression models were performed, to determine the predictive validity of the indicators, with the summary score as the dependent variable. In the first model, the self-regulation questionnaire was taken as an independent variable. In the second, the independent variable of summary performance time was added, and in the third, the TA categories were also added. The results are shown in Table 6.
Finally, to identify student profiles according to the use of self-regulatory processes, an analysis of four clusters was performed. One of them, cluster 2, was formed by a single subject, who used only three strategies, and all three were from the same category (stimulate interest). Cluster 1 (N = 3) differed from the rest of the clusters due to the value of the strategy of relating to background knowledge. Cluster 3 (N = 17) differed by the use of goal-setting and help-seeking. Meanwhile, cluster 4 was characterized by the use of monitoring and self-instructions. Results are shown in Table 7.
A t test was performed to relate the clusters with performance time and RC performance score. Clusters 1 and 2 were discarded because they had three and one student respectively. For RC score, results were t(21) = − 1.72, p = 0.09 and for task performance time t(84) = − 1.99, p = 0.05. Cluster 4 students tend to have better performance than cluster 3 students and use more time.
Discussion and conclusions
Discussion
The first objective of this paper is to describe the use of self-regulated processes in text comprehension tasks by students in the last year of primary education. The students’ perception of the use of self-regulated processes of Zimmerman`s cyclical model measured by the questionnaire was high and did not have much relation with TA data. This is consistent with previous studies (Heirweg et al., 2019; Rogiers et al., 2020). Although both the perceived self-regulation and self-efficacy of these students were high, the mean performance in the task was below half the maximum score. This shows some bias in the data obtained through self-reporting: students perceive themselves as more competent than what they show in their performance.
In general terms, the information obtained through TA shows that the students focused on the use of processes from the performance phase, in particular monitoring and the use of self-instructions. In the second place, the students used processes corresponding to the forethought phase, mainly setting goals and strategic planning. Self-reflection processes were used sparingly, and self-evaluation was the most used process in this phase. This data is consistent with Heirweg et al. (2019). Reading tasks can encourage the use of strategies in the performance phase over the rest of the strategies. The task proposed to the students led them directly to reading the text and writing the summary. Regarding the processes with a motivational component, such as self-efficacy or the stimulation of interest, they were the least used. The task was considered to be of high difficulty, which led to reactions of motivational nature such as complaints, but the use of processes to self-regulate motivation was scarce.
The second objective was methodological and consisted of estimating the convergent validity between three assessment methods of self-regulation. The self-regulation questionnaire and the PVA measure the processes of Zimmerman’s model. In general, the relationships between TA, the questionnaire and the traces were low, and the relationships between the traces and the TA were moderate. This low convergence between the TA and the questionnaire has already been established by previous research (Cirino et al., 2017; Heirweg et al., 2019; Rogiers et al., 2020). Questionnaires evaluate students’ perception of their use of processes, while TA evaluates students’ actual behaviour.
Regarding traces, the most discriminant measure, and the one that had the highest correlation with the questionnaire and the TA, was task performance time. The students who used more time made use of a greater number of strategies. They also had a better perception of the use of self-regulation, according to the questionnaire.
The third objective of this study is to estimate the predictive validity regarding text comprehension performance. In this respect, it was found that the questionnaire only explains 5% of the variance in the summary quality. Meanwhile, if TA and performance time measures are considered along with the questionnaire, these factors explain 40% of the variance in the summary result.
This study confirms previous research indicating that the best-performing students in RC use more self-regulation strategies (Heirweg et al., 2020), as a correlation was found between task performance and the number of strategies used, measured through the TA protocol. Concerning the self-regulation strategies that were linked to performance in the different phases, control is one of the factors that appear related to success. This involved students noticing that they needed to read again, change their answers or reformulate what they had written. As Eme et al. (2006) pointed out, as the task becomes more complex, control and monitoring may turn into self-instructions, which in this study have been shown to explain in part good performance. Using this strategy, students asked themselves what something meant, told themselves what they should reread, calmed themselves down or gave themselves explanations to make decisions. Another factor related to text comprehension was specific task strategies. These are reading strategies such as underlining, analysing titles, memorizing and making inferences. These results are consistent with Heirweg et al. (2020) and Wang (2016), who found that cognitive-type strategies and specific reading strategies characterized the best-performing students.
Among the traces, the indicator that showed a stronger relationship with task performance was performance time. The time that the students dedicated to completing the task was highly variable. In general, the best performers were those who, in order to prepare a good summary of the text, resorted to rereading and searching for specific information in the text, which implies the use of more time.
Different student profiles were identified in terms of their use of self-regulated processes. A small group of students used mainly the strategy of relating what they read with their previous knowledge. This was rarely used in the sample in general. Another student profile predominantly employed goal-setting during the forethought phase, contemplating their objectives for the task or recalling the task instructions. Finally, there was a group of students who were characterized by the use of the performance phase strategies: monitoring and self-instructions. This last group used more time to complete the task and showed a tendency to obtain better outcomes.
Conclusions
The results of this work are relevant to the field of self-regulation research, as well as to educational practice. This work seeks to further current knowledge about ways of assessing self-regulated learning. In this study, the think-aloud method was useful to describe the use of self-regulated processes by primary school students and to predict performance in a reading task. This methodology allowed to describe the best-performing students and the processes they used. Regarding self-regulation measured by means of a questionnaire, it did not offer information that correlated with the TA and showed a very weak relationship with task performance. Finally, the analysis of some TT was also relevant to explaining performance and proved to be an interesting methodology to include, in particular the consideration of task performance time.
It should be noted that the RC assessment task in this study was designed based on texts and curricular tasks used in real classrooms. This gives it greater ecological validity, compared to other studies that use standardized texts, which are not usually used in teaching. However, in future instances, it would be necessary to assess students’ background knowledge on the text’s topic.
In terms of practical implications, the analysis of the factors that were related to task performance was used as an input for another study with the same sample. The assessment methodologies applied in this work were replicated to evaluate the impact of an intervention. Implications for educational practice include the use of thinking aloud as a tool for teachers to familiarize themselves with the theoretical categories of self-regulation. Procedures like PVA aid in identifying self-regulated processes, enhancing comprehension and consequently facilitating effective teaching.
There were several limitations of this study. Firstly, the number of participants in this study was limited. It should be noted that the use of think-aloud protocols involves the evaluation of individual students in sessions that averaged 30 min and the subsequent transcription and analysis of these sessions. This means that, in general, this type of study is carried out with samples that do not exceed 100 participants. Regarding the low presence of motivational processes, this could be related to the use of Zimmerman’s model for coding. In this model, the higher presence of motivational processes is found in the forethought phase, within self-motivation beliefs. Perhaps, if students had spent more time in the forethought phase, other processes related to self-regulation of motivation would have emerged. Furthermore, it is worth noting that motivational verbalizations are not recurrent, whereas other types of verbalizations, such as those used to monitor task execution, are. For example, if students expressed their interest in the text’s topic, they vocalized it once. In contrast, other processes like task monitoring were verbalized repeatedly throughout the task. Therefore, self-motivation processes will be less frequently utilized when compared quantitatively to other processes.
Regarding the questionnaire used, it presented discrepancies with the think-aloud protocol, and this could be due to the typical drawbacks of this type of instrument: the desirability bias of students, their difficulties in making the strategies explicit after the task or difficulties in understanding what the questionnaire asked them. In the TA, the processes are explicated spontaneously with their own words during the task. In the questionnaire, students are required to recall what they did and respond based on pre-written descriptions. This appears to be more complex. Another limitation of the questionnaire is that it measures self-regulation in a global manner. The search for an instrument that measures the different processes and subprocesses should continue.
Finally, it should be noted that evaluating students in a single instance and using a single text may limit the results.
It is expected to shed some light on the problem of evaluating the control and regulation of human action.
References
American Educational Research Association (Ed.). (2011). Report and recommendations for the reauthorization of the institute of education sciences. American Educational Research Association.
Balbi, A., Cuadro, A., Trías, D. (2009). Comprensión lectora y reconocimiento de palabras. Ciencias Psicológicas, 3(2), 153–160.
Beek, M., Opdenakker, M., Spijkerboer, A. W., Brummer, L., Ozinga, H. W., & Strijbos, J. (2019). Scaffolding expository history text reading: Effects on adolescents’ comprehension, self-regulation, and motivation. Learning and Individual Differences, 74, 101749. https://doi.org/10.1016/j.lindif.2019.06.003
Burin, D. I., Gonzalez, F. M., Barreyro, J. P., & Injoque-Ricle, I. (2020). Metacognitive regulation contributes to digital text comprehension in E-learning. Metacognition and Learning, 15(3), 391–410. https://doi.org/10.1007/s11409-020-09226-8
Butler, D. L., & Schnellert, L. (2015). Success for students with learning disabilities: What does self-regulation have to do with it? In T. Cleary (Ed.), Self-regulated learning interventions with at-risk youth: Enhancing adaptability, performance, and well-being (pp. 89–111). American Psychological Association. https://doi.org/10.1037/14641-005
Cirino, P. T., Miciak, J., Gerst, E., Barnes, M. A., Vaughn, S., Child, A., & Huston-Warren, E. (2017). Executive function, self-regulated learning, and reading comprehension: A training study. Journal of Learning Disabilities, 50(4), 450–467. https://doi.org/10.1177/0022219415618497
Cuadro, A., Costa, D., Trías, D., Ponce de León, P., & Daset, L. R. (2009). Evaluación del Nivel Lector.: Manual Técnico del Tests de Eficacia Lectora (TECLE). Ciencias Psicológicas, 3(2), 247–248. https://doi.org/10.22235/cp.v3i2.156
Curione, K., Gründler, V., Píriz, L., & Huertas, J. A. (2017). MSLQ-UY, validación con estudiantes universitarios uruguayos. Revista Evaluar, 17(2), 1–17. https://doi.org/10.35670/1667-4545.v17.n2.18716
de Guerrero, M. (2018). Going covert: Inner and private speech in language learning. Language Teaching, 51(1), 1–35. https://doi.org/10.1017/S0261444817000295
de Milliano, I., van Gelderen, A., & Sleegers, P. (2016). Types and sequences of self-regulated reading of low-achieving adolescents in relation to reading task achievement. Journal of Research in Reading, 39(2), 229–252. https://doi.org/10.1111/1467-9817.12037
Dent, A. L., & Koenka, A. C. (2016). The relation between self-regulated learning and academic achievement across childhood and adolescence: A meta-analysis. Educational Psychology Review, 28(3), 425–474. https://doi.org/10.1007/s10648-015-9320-8
Denton, C., Enos, M., York, M. J., Francis, D. J., Barnes, M. A., Kulesz, P., Fletcher, J., & Carter, S. (2015). Text-processing differences in adolescent adequate and poor comprehenders reading accessible and challenging narrative and informational text. Reading Research Quarterly, 50(4), 393–416. https://doi.org/10.1002/rrq.105
Dignath, C., & Büttner, G. (2008). Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacognition and Learning, 3(3), 231–264. https://doi.org/10.1007/s11409-008-9029-x
Dignath, C., & Veenman, M. V. J. (2020). The role of direct strategy instruction and indirect activation of self-regulated learning-evidence from classroom observation studies. Educational Psychology Review, 33(2), 489–533. https://doi.org/10.1007/s10648-020-09534-0
Eme, E., Puustinen, M., & Coutelet, B. (2006). Individual and developmental differences in reading monitoring: When and how do children evaluate their comprehension? European Journal of Psychology of Education, 21(1), 91–115. https://doi.org/10.1007/BF03173571
Ferrer, A., Vidal-Abarca, E., Serrano, M. Á, & Gilabert, R. (2017). Impact of text availability and question format on reading comprehension processes. Contemporary Educational Psychology, 51, 404–415. https://doi.org/10.1016/j.cedpsych.2017.10.002
Harding, S. M., English, N., Nibali, N., Griffin, P., Graham, L., Alom, B., & Zhang, Z. (2019). Self-regulated learning as a predictor of mathematics and reading performance: A picture of students in grades 5 to 8. Australian Journal of Education, 63(1), 74–97. https://doi.org/10.1177/0004944119830153
Heirweg, S., De Smul, M., Devos, G., & Van Keer, H. (2019). Profiling upper primary school students’ self-regulated learning through self-report questionnaires and think-aloud protocol analysis. Learning and Individual Differences, 70, 155–168. https://doi.org/10.1016/j.lindif.2019.02.001
Heirweg, S., De Smul, M., Merchie, E., Devos, G., & Van Keer, H. (2020). Mine the process: Investigating the cyclical nature of upper primary school students’ self-regulated learning. Instructional Science, 48(4), 337–369. https://doi.org/10.1007/s11251-020-09519-0
Hu, J., & Gao, X. (2017). Using think-aloud protocol in self-regulated reading research. Educational Research Review, 22, 181–193. https://doi.org/10.1016/j.edurev.2017.09.004
León, J. A., Moreno, J. D., Arnal, L. A., Escudero, I., & Olmos, R. (2015). Baremación De una prueba estandarizada de resúmenes (RESUMeV) para Los Niveles De 4º Y 6º De educación primaria. Clínica Y Salud, 26(1), 49–55. https://doi.org/10.1016/j.clysa.2015.01.001
Máñez, I., Vidal-Abarca, E., & Magliano, J. P. (2022). Comprehension processes on question-answering activities: A think-aloud study. 1-Electronic Journal of Research in Educational Psychology, 20(56), 1–26. https://doi.org/10.25115/ejrep.v20i56.3776
Merchie, E., Van Keer, H., & Vandevelde, S. (2014). Development of the text-learning strategies inventory: Assessing and profiling learning from texts in fifth and sixth grade. Journal of Psychoeducational Assessment, 32(6), 533–547. https://doi.org/10.1177/0734282914525155
Minguela, M., Solé, I., & Pieschl, S. (2015). Flexible self-regulated reading as a cue for deep comprehension: Evidence from online and offline measures. Reading and Writing, 28(5), 721–744. https://doi.org/10.1007/s11145-015-9547-2
Moos, D. C., & Ringdal, A. (2012). Self-regulated learning in the classroom: A literature review on the teacher’s role. Education Research International, 2012, 1–15. https://doi.org/10.1155/2012/423284
Ochoa, E. M., Río, N. E., Mellone, C., & Simonetti, C. E. (2019). Eficacia De Un programa de mejora de la comprensión lectora en educación primaria. Revista De Psicología Y Educación, 14(1), 63–73. https://doi.org/10.23923/rpye2019.01.172
Okkinga, M., van Steensel, R., van Gelderen, A. J., & Sleegers, P. J. (2018). Effects of reciprocal teaching on reading comprehension of low-achieving adolescents. The importance of specific teacher skills. Journal of Research in Reading, 41(1), 20–41. https://doi.org/10.1111/1467-9817.12082
Oliver, E., Markland, D., Hardy, J., & Petherick, C. (2008). The effects of autonomy-supportive versus controlling environments on self-talk. Motivation & Emotion, 32(3), 200–212. https://doi.org/10.1007/s11031-008-9097-x
Panadero, E., & Alonso-Tapia, J. (2014). ¿Cómo autorregulan nuestros alumnos? Modelo De Zimmerman sobre estrategias de aprendizaje. Annals of Psychology, 30(2), 450–462. https://doi.org/10.6018/analesps.30.2.167221
Panadero, E., Klug, J., & Järvelä, S. (2016). Third wave of measurement in the self-regulated learning field: When measurement and intervention come hand in hand. Scandinavian Journal of Educational Research, 60(6), 723–735. https://doi.org/10.1080/00313831.2015.1066436
Perry, N., Thauberger, C., & Hutchinson, L. (2010). GStudy traces of children’s self-regulated learning in the Lifecycles Learning Kit. Psychological Test and Assessment Modeling, 52(4), 432–453.
Rogiers, A., Merchie, E., & Van Keer, H. (2019). Learner profiles in secondary education: Occurrence and relationship with performance and student characteristics. The Journal of Educational Research, 112(3), 1–12. https://doi.org/10.1080/00220671.2018.1538093
Rogiers, A., Merchie, E., & Van Keer, H. (2020). What they say is what they do? Comparing task-specific self-reports, think-aloud protocols, and study traces for measuring secondary school students’ text-learning strategies. European Journal of Psychology of Education, 35(2), 315–332. https://doi.org/10.1007/s10212-019-00429-5
Rovers, S. F., Clarebout, G., Savelberg, H. H., de Bruin, A. B., & van Merriënboer, J. J. (2019). Granularity matters: Comparing different ways of measuring self-regulated learning. Metacognition and Learning, 14(1), 1–19. https://doi.org/10.1007/s11409-019-09188-6
Saat, F., & Özen, E. G. (2022). Effect of self-evaluation-based oral reading method in elementary school on reading fluency and reading comprehension. Participatory Educational Research, 9(4), 437–462. https://doi.org/10.17275/per.22.48.9.2
Steiner, M., Loon, M. H., Van, Bayard, N. S., & Roebers, C. M. (2020). Development of children’ s monitoring and control when learning from texts: Effects of age and test format. Metacognition and Learning, 15, 3–27. https://doi.org/10.1007/s11409-019-09208-5
Stoeger, H., & Zeidner, M. (2019). Self-regulated learning in gifted, talented, and high-achieving learners. High Ability Studies, 30(1–2), 1–8. https://doi.org/10.1080/13598139.2019.1601326
Tabullo, ÁJ., Wainselboim, A. J., & París, L. A. (2022). La comprensión de textos expositivos y narrativos y su relación con El Vocabulario. La Eficacia Lectora Y El control inhibitorio en niños de cuarto grado. Traslaciones Revista latinoamericana de Lectura y Escritura, 9(17), 39–59.
Van Ammel, K., Aesaert, K., de Smedt, F., & van Keer, H. (2021). Skill or will? The respective contribution of motivational and behavioural characteristics to secondary school students’ reading comprehension. Journal of Research in Reading, 44(3), 574–596. https://doi.org/10.1111/1467-9817.12356
van de Pol, J., de Bruin, A. B., van Loon, M. H., & van Gog, T. (2019). Students’ and teachers’ monitoring and regulation of students’ text comprehension: Effects of comprehension cue availability. Contemporary Educational Psychology, 28, 236–249. https://doi.org/10.1007/s11145-015-9547-2
Van Der Graaf, J., Lim, L., Fan, Y., Kilgour, J., Moore, J., Bannert, M., Gasevic, D., & Molenaar, I. (2021). Do instrumentation tools capture self-regulated learning? In LAK21: 11th international learning analytics and knowledge conference (pp. 438–448). Association for Computing Machinery. https://doi.org/10.1145/3448139.3448181
Van Halem, N., Van Klaveren, C., Drachsler, H., Schmitz, M., & Cornelisz, I. (2020). Tracking patterns in self-regulated learning using students’ self-reports and online trace data. Frontline Learning Research, 8(3), 140–163. https://doi.org/10.14786/flr.v8i3.497
Wang, Y. H. (2016). Reading strategy use and comprehension performance of more successful and less successful readers: A think-aloud study. Educational Sciences: Theory & Practice, 16(5), 1789–1813.
Wu, L., Valcke, M., & Van Keer, H. (2021). Supporting struggling readers at secondary school: An intervention of reading strategy instruction. Reading and Writing, 34(8), 2175–2201. https://doi.org/10.1007/s11145-021-10144-7
Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into Practice, 41(2), 64–70. https://doi.org/10.1207/s15430421tip4102_2
Zimmerman, B. J. (2011). Motivational sources and outcomes of self-regulated learning and performance. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of Self-Regulation of Learning and Performance (pp. 63–78). Routledge.
Acknowledgements
Agencia Nacional de Investigación e Innovación del Uruguay. Scholarship for PhDs Abroad.
Funding
Open Access funding provided thanks to the CRUE-CSIC agreement with Springer Nature. This work was supported by Convocatoria del plan nacional de I+i. Ministerio de Ciencias y Universidades. Retos Ministerio de Economía y Competitividad de España, Desarrollado en: UAM. Ref EDU2017-89036-P.Ref PID2022-138175NB-I00.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Valentina Ronqui Leites, Daniel Trías Seferian and Juan Antonio Huertas Martínez. The first draft of the manuscript was written by Valentina Ronqui Leites, and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
The investigation was approved by the Ethics Committees of Universidad Católica del Uruguay and Universidad Autónoma de Madrid. All the students’ families submitted their free informed consent, and the students agreed to participate.
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Valentina Ronqui Leites. Universidad Autónoma de Madrid. valentina.ronqui@estudiante.uam.es
Current themes of research:
Self-regulated learning. Reading. Teacher professional development
Most relevant publications in the field of Psychology of Education:
Ronqui, V., Trías, D., Huertas, J.A., (In press) Creencias y prácticas de maestras de Educación Primaria acerca de la autorregulación del aprendizaje (Beliefs and practices of primary school teachers about self-regulated learning). Interdisciplinaria Revista de Psicología y Ciencias afines.
Trias, D., Huertas, J. A., Mels, C., Castillejo, I., & Ronqui, V. (2021). Autorregulación en el aprendizaje, desempeño académico y contexto socioeconómico al finalizar la escuela primaria (Self-regulation in learning, academic performance, and socioeconomic context at the end of primary school).Interamerican Journal of Psychology 55(2).
Ronqui, V., Sánchez, M. F., & Trías, D. (2021). La enseñanza de la autorregulación en aulas de educación primaria (Teaching self-regulated learning in primary school classrooms.). Cuadernos de investigación Educativa, 12(2), 5-22.
Daniel Trías Seferian. Education Department, Universidad Católica del Uruguay, Montevideo, Uruguay. gtrias@ucu.edu.uy
Current themes of research:
Self-regulated learning. Motivation. Reading. Teacher professional development
Most relevant publications in the field of Psychology of Education:
Trías, D., & Carbajal, M. (2020). Adaptación de una escala de procrastinación académica para adolescentes de educación secundaria (Adaptation of an academic procrastination scale for secondary school adolescents). Avances en Psicología Latinoamericana, 38(3), 36-51.
Trias, D., Mels, C., & Huertas, J. A. (2021). Teaching to Self-Regulate in Mathematics: A Quasi-Experimental Study with Low-Achieving Elementary School Students. Revista electrónica de investigación educativa, 23.
Trias, D., Huertas, J. A., Mels, C., Castillejo, I., & Ronqui, V. (2021). Autorregulación en el aprendizaje, desempeño académico y contexto socioeconómico al finalizar la escuela primaria (Self-regulation in learning, academic performance, and socioeconomic context at the end of primary school). Interamerican Journal of Psychology 55(2).
Juan Antonio Huertas Martínez. Basic Psychology Department, Universidad Autónoma de Madrid, Madrid, Spain. juanantonio.huertas@uam.es
Current themes of research:
Self-regulated learning. Motivation. Teacher professional development. Developmental disabilities
Most relevant publications in the field of Psychology of Education:
Alonso-Tapia, J., Merino-Tejedor, E. & Huertas, J. A. (2022). Academic engagement: assessment, conditions, and effects-a study in higher education from the perspective of the person-situation interaction. European Journal of Psychology of Education, 1-25.
Domínguez, A. Q., Ruiz, M. Á., Huertas, J. A., & Alonso-Tapia, J. (2020). Development and validation of the School Climate Questionnaire for Secondary and High School Teachers (SCQ-SHST). Anales De Psicología/Annals of Psychology, 36(1), 155-165.
Alonso-Tapia, J., Ruiz, M. Á., & Huertas, J. A. (2020). Differences in classroom motivational climate: Causes, effects and implications for teacher education. A multilevel study. Anales De Psicología/Annals of Psychology, 36(1), 122-133.
Nieto, C., & Huertas, J. A. (2020). Stereotypies and restricted and repetitive behaviours in autism. Journal for the Study of Education and Development, 43(4), 824-840.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Leites, V.R., Seferian, D.T. & Martínez, J.A.H. Assessing self-regulated processes: what do primary school students do, say and think in the process of understanding a text?. Eur J Psychol Educ (2024). https://doi.org/10.1007/s10212-024-00830-9
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1007/s10212-024-00830-9