How Does Prior Knowledge Influence Eye Fixations and Sequences of Cognitive and Metacognitive SRL Processes during Learning with an Intelligent Tutoring System?

Abstract

The goal of this study was to use eye-tracking and log-file data to investigate the impact of prior knowledge on college students’ (N = 194, with a subset of n = 30 for eye tracking and sequence mining analyses) fixations on (i.e., looking at) self-regulated learning-related areas of interest (i.e., specific locations on the interface) and on the sequences of engaging in cognitive and metacognitive self-regulated learning processes during learning with MetaTutor, an Intelligent Tutoring System that teaches students about the human circulatory system. Results revealed that there were no significant differences in fixations on single areas of interest by the prior knowledge group students were assigned to; however there were significant differences in fixations on pairs of areas of interest, as evidenced by eye-tracking data. Furthermore, there were significant differences in sequential patterns of engaging in cognitive and metacognitive self-regulated learning processes by students’ prior knowledge group, as evidenced from log-file data. Specifically, students with high prior knowledge engaged in processes containing cognitive strategies and metacognitive strategies whereas students with low prior knowledge did not. These results have implications for designing adaptive intelligent tutoring systems that provide individualized scaffolding and feedback based on individual differences, such as levels of prior knowledge.

Introduction

Self-regulated learning (SRL) is an educational construct that defines students as active participants during learning by engaging in planning, monitoring, and strategy-use, as opposed to passively being recipients of information (Azevedo et al. 2018; Winne and Azevedo 2014). For example, instead of students listening to a class lecture, they are given a learning task, and plan the content pages they need to read to learn the material, monitor how their learning is progressing by assessing if they understand the content or if they should re-read it, and use learning strategies that help them learn the material, such as taking notes or summarizing the content into their own words. As such, successful SRL involves students engaging in cognitive and metacognitive processes, while keeping track of their emotions and motivation during learning (Azevedo et al. 2018).

By having students self-regulate their own learning, this can enhance their learning of complex topics because they are participating in their learning processes, which is why SRL is a key skill for all students (Winne and Azevedo 2014). However, research has shown that students do not typically deploy these strategies effectively (or at all) during learning (Azevedo 2014), which led to the development of advanced learning technologies, such as intelligent tutoring systems, game-based learning environments, hypermedia and multimedia environments, and simulations (Azevedo et al. 2018; Biswas et al. 2016; Graesser 2013; Lester et al. 2013; Nietfeld 2018). These different types of learning technologies were designed to foster SRL in different ways. For example, intelligent tutoring systems foster the use of SRL processes by interacting with students and providing them with scaffolding based on their performance, while game-based learning environments foster learning while ensuring students are highly motivated and engaged during learning. No matter what the specific elements are, most advanced learning technologies serve the same overall purpose – to foster effective SRL during learning about complex topics.

Theoretical Framework

For our theoretical framework, we use Winne & Hadwin’s information processing theory of SRL (Winne and Hadwin 1998, 2008; Winne 2018), which states that learning occurs through a series of four cyclical phases, where SRL is an event that unfolds over time. Specifically, the four phases involve: (1) understanding the task, (2) setting goals and making plans for accomplishing them, (3) using the learning strategies planned out in the previous phase, and (4) making adaptations to the goals and plans, and learning strategies. The model explains that these four phases are cyclical in nature, however they are not necessarily sequential, such that students can engage in these phases in any order they like, and can engage in multiple phases simultaneously. For example, a student engaging in a learning strategy might choose to make an adaptation to that strategy in real-time. They can determine activating their prior knowledge does not seem to be a useful strategy because they do not know anything about the topic, so they decide to read the content and coordinate information from the text and diagram together instead. Additionally, the model posits that there is a set of cognitive conditions, which can impact how students progress through all four phases. An example cognitive condition is prior domain knowledge, meaning that based on the amount of prior knowledge students have, this can impact their ability to understand the task, set meaningful goals and plans, use sophisticated learning strategies, and make adaptations to engage in SRL strategies during learning. This model is appropriate for our study because we view SRL as an event that occurs during learning, and we investigated the role of prior knowledge on the use of these SRL strategies during learning with MetaTutor.

MetaTutor: A Hypermedia-Based ITS

MetaTutor is a hypermedia-based intelligent tutoring system that teaches participants about the human circulatory system (Azevedo et al. 2013, 2018). The environment contains 47 pages of text content and static diagrams, and gives participants the overall learning goal of learning as much as they can about the human circulatory system in 90 min. The MetaTutor environment was designed specifically to foster SRL processes during learning (see Fig. 1). At the top left corner of the interface in Fig. 1, there is a timer, which counts down the amount of time remaining in the 90-min learning session; and below the timer is a table of contents where participants can select the content page they want to read. The text content is in the center with the corresponding diagram to its right. The overall learning goal¹ is at the top center, with the sub-goals just below it. Prior to the 90 min allotted to learning, participants engaged in a sub-goal setting phase that fosters planning. This was done before the 90 min began counting down because we did not want this process to impact the amount of time participants spent interacting with the MetaTutor content, as some participants took longer to set their sub-goals than others. During this planning phase, participants set two sub-goals (e.g., heartbeat, purposes of the circulatory system) with Pam the Planner, one of the pedagogical agents (for more information about the sub-goal setting phase, see Harley et al. 2017). There are seven pre-set sub-goals: ([1] path of blood flow, [2] heartbeat, [3] heart components, [4] blood vessels, [5] blood components, [6] purposes of the circulatory system, and [7] malfunctions of the circulatory system), and participants were guided toward setting two sub-goals during the sub-goal setting phase, and could set up to five more during learning.

Fig. 1

Screenshot of the MetaTutor interface with notebook open (right)

In the upper right corner of Fig. 1, there is the pedagogical agent. MetaTutor has four pedagogical agents where each agent is responsible for a different aspect of SRL. Gavin the Guide steers participants through the environment by introducing MetaTutor, providing participants with introductory videos, administering self-reports, and administering the pre- and post-tests. Pam the Planner assists with planning, and therefore helps participants set sub-goals and activate their prior knowledge. Sam the Strategizer helps participants engage in cognitive learning strategies, such as taking notes, making summaries, and making inferences. Mary the Monitor assists participants with their use of metacognitive monitoring processes (defined as processes that involve participants monitoring their learning), such as judging how well they understand the content they are reading (judgment of learning [JOL]), how familiar the material seems (feeling of knowing [FOK]), how relevant the material is to their active sub-goal (content evaluation [CE]), and have they read through enough material to proceed to the next sub-goal (monitoring progress towards goals [MPTG]). Each agent is present one at a time, depending on the activity the participant is doing. For example, if the participant is engaging in metacognitive monitoring, Mary will be present on the interface (as shown in Fig. 1). Mary will be present if the strategy was prompted, or if the participant self-initiated the strategy by clicking on the SRL palette. If participants are prompted or self-initiate making summaries or inferences, Sam the Strategizer will be present. If participants are setting sub-goals (during the sub-goal setting phase or during their 90 min of interacting with MetaTutor) or activating their prior knowledge, Pam will be present. Gavin is present to administer self-reports and the pre- and post-tests. Therefore, if a participant is not prompted (see Experimental Conditions section) or does not initiate any SRL processes, Gavin will remain on the screen during the 90 min, while Pam will always be present during the sub-goal setting phase. Below the agent is the SRL palette, which participants can use to self-initiate the use of cognitive (e.g., take notes, summarize) and metacognitive (e.g., judgment of learning, content evaluation) SRL processes. Therefore, based on these interface elements, participants have the opportunity to engage in different cognitive and metacognitive SRL processes during learning with MetaTutor.

Literature Review

For this study, we used multichannel data as well as educational data mining techniques to investigate how prior knowledge impacted learning with MetaTutor (see above). There have been many studies that have used eye-tracking data to investigate learning of various topics (e.g., Gerjets et al. 2011; Inglis and Alcock 2012; Mason et al. 2017; Scheiter and Eitel 2016); however these studies have not specifically investigated SRL during learning with ITSs. Here, we discuss how studies have investigated SRL during learning with MetaTutor, where studies by Bondareva et al. (2013) and Jaques et al. (2014) specifically examined SRL during learning with MetaTutor using eye-tracking data. Bondareva et al. (2013) were able to successfully predict learning gain using a simple logistic regression with gaze features (e.g., rate of fixations, duration of fixations) and the proportion of transitions between different areas of interest (AOIs). They were able to use these features to predict learning gain with 78.3% accuracy. Jaques et al. (2014) used transitions between AOI pairs as a measure of engagement. Using this method, they were successfully able to use these transitions to predict students’ levels of boredom and curiosity, with 69% and 73% accuracy, respectively. Therefore, these are two specific examples investigating emotional states and SRL with MetaTutor, in addition to many studies using eye-tracking data within ITSs to predict emotional states and overall learning with these systems (e.g., Conati et al. 2013; D’Mello 2016; D’Mello et al. 2012).

In addition to using traditional statistical techniques such as ANOVAs and regressions, researchers have been using educational data mining techniques (e.g., Baker and Yacef 2009; Biswas et al. 2018) to investigate how students are engaging in sequences of learning-related behaviors during learning with ITSs. Specific to SRL, researchers have applied techniques such as cluster analysis (Bouchet et al. 2013), and differential sequence mining (Kinnebrew et al. 2013, the technique used for this study. In a study conducted by Bouchet et al. (2013), they did a cluster analysis with data from students who learned using MetaTutor, where they clustered students into 3 separate clusters (based on a series of SRL-related variables; see Bouchet et al. 2013), and defined these clusters (0, 1, and 2) having 3 distinct profiles. Group 2 (high performance monitoring) had the highest performance on quizzes, checked their notes more, and set the most sub-goals. Group 1 (low performance reading) had the lowest performance and spent the most time reading. Group 0 (mid point note-taking) performed in the middle of groups 1 and 2, and took the most notes. Therefore, by clustering students, they were able to identify students with different types of learning profiles (i.e., different uses of cognitive and metacognitive SRL processes) who were learning with MetaTutor. Kinnebrew et al. (2013) used differential sequence mining to statistically compare the frequency of occurrence of sequential patterns between high and low performing students as they learned and created concept causal maps with Betty’s Brain, a teachable agents ITS. Results revealed significant differences in the patterns of activities relating to productive vs. unproductive sequences for high vs. low performing students (Kinnebrew et al. 2013). Therefore, this study was able to identify statistically significant differences in sequences of using different SRL processes during learning, and how these sequences differed based on students’ levels of performance.

Although these studies have used multichannel data and advanced statistical techniques to study SRL and learning with ITSs, few studies have also investigated the impact of different types of individual differences (e.g., goal orientation, personality; Lallé et al. 2017) on SRL with these ITSs. For example, prior knowledge has been found to significantly impact learning with ITSs and hypermedia-learning environments (Greene et al. 2010; Moos and Azevedo 2008; Taub et al. 2014; Trevors et al. 2014). Greene et al. (2010) used structural equation modeling with think-aloud data to investigate the relationship between prior knowledge, theory of intelligence, SRL, and performance during learning with a hypermedia-learning environment. Their results revealed that SRL acted as a significant modifier between prior knowledge and performance, and theory of intelligence and performance, such that there was a positive relationship between prior knowledge and performance, which was further enhanced with effective SRL (Greene et al. 2010), emphasizing the important influence of both prior knowledge and SRL on overall learning with ITSs. In a second study using think-aloud data, Moos and Azevedo (2008) found that when students had high prior knowledge, this was positively related to their use of planning and monitoring SRL processes, and negatively related to their use of strategizing SRL processes, again revealing the important influence of prior knowledge on the use of different types of SRL processes.

Furthermore, specific studies have been done investigating the impact of prior knowledge on the use of SRL processes during learning with MetaTutor. Taub et al. (2014) investigated the frequency of use of cognitive and metacognitive SRL processes and found that there were significant differences in the use of metacognitive, but not cognitive SRL processes between prior knowledge groups. In addition, they found that there were different patterns of using cognitive and metacognitive processes between prior knowledge groups; however no statistical analyses were conducted on the patterns. In a study conducted by Trevors et al. (2014), they investigated note taking during learning with MetaTutor by their prior knowledge group, and determined that when students were provided with prompts and feedback, students with low prior knowledge took significantly more notes than students with high prior knowledge, emphasizing not only the important role of prior knowledge on the use of SRL processes, but also how different types of feedback may play a role as well.

Many studies have investigated the impact of prior knowledge on SRL, where most of these studies have predominantly used traditional statistical techniques. Many studies have used eye-tracking data as well as sequence mining to investigate SRL during learning with ITSs; however few of these studies have included the impact of prior knowledge on SRL, and the AOIs investigated are not typically categorized by SRL-related variables. In addition, when investigating sequences of SRL-related activities, activities are not typically coded by accuracy, such as the accuracy of a metacognitive judgment or cognitive learning strategy. Therefore, the goal of this study was to investigate the impact of prior knowledge on fixations on SRL-related AOIs and on the sequences of engaging in accurate or inaccurate cognitive and metacognitive SRL processes during learning with MetaTutor, an ITS. In addition, we seek to demonstrate how we can use multichannel data to assess SRL processes, where we view SRL as an event that temporally unfolds over time, with implications for using these types of data to design adaptive ITSs for different types of learners.

Current Study

The current study is based on our prior work (Taub and Azevedo 2016), investigating the impact of prior content and prior sub-goal knowledge on students’ proportions of fixations on different AOIs during learning with MetaTutor. Our results did not reveal many significant differences in proportions of fixations, where we only found a significant difference in proportions of fixations on diagrams between prior sub-goal knowledge groups. Students with low prior sub-goal knowledge had significantly longer proportions fixating on the diagram than students with high prior sub-goal knowledge. We did find significant differences in AOI pairs between the prior knowledge groups, which we used as a basis for running sequence-mining techniques.

The goal of this study was to extend our original analyses (Taub and Azevedo 2016) on how levels of prior knowledge influenced proportions of fixations on AOIs related to SRL.

We extend these findings by presenting new analyses where we examined sequences of engaging in cognitive and metacognitive SRL processes during learning, including the accuracy of these SRL processes. In doing so, our goal is to not only examine differences in fixations between different levels of prior knowledge and sequences of use of SRL processes between different levels of prior knowledge, but also to demonstrate how we used different types of process data (eye tracking and log files) and different statistical techniques (ANCOVA/MANCOVA and sequence mining) to examine how participants with different levels of prior knowledge engaged in SRL during learning with MetaTutor. We examined the proportions of fixations on different AOIs to account for different session times for participants. We grouped AOIs into 3 categories: learning-related AOIs (text content, diagram, notes, and table of contents), SRL-related AOIs (timer, SRL palette, sub-goals, and learning goal), and the pedagogical agent AOI. AOI pairs included text content as reading text is the foundation for knowledge acquisition and SRL, such that students predominantly spend time reading the text content, and therefore we make the assumption that SRL processes can be examined from AOI pairs that began with text content. In addition, for investigating sequences of engaging in cognitive and metacognitive SRL processes, we examined each instance of self-initiating the use of these processes, and coded them as being cognitive vs. metacognitive, and accurate vs. inaccurate, yielding a series of four different codes. We used these variables to form research questions and hypotheses.

Research Questions and Hypotheses

We addressed the following research questions: (RQ1): Are there significant differences in proportional learning gain by prior knowledge group, while controlling for condition?; (RQ2): Are there significant differences in fixation behavior by prior knowledge group, while controlling for condition?; (RQ3): Can we detect sequences of engaging in cognitive and metacognitive SRL processes by prior knowledge group?; and (RQ4): Are there significant differences in sequences of cognitive and metacognitive SRL processes by prior knowledge group, while controlling for condition?

We hypothesized that:

• (H1) There will be significant differences in proportional learning gain by prior knowledge group, such that while controlling for condition, participants with high prior knowledge will have significantly higher proportional learning gains than participants with low prior knowledge;

• (H2) There will be significant differences in the proportions of time spent and frequencies of fixations on AOIs, while controlling for condition, based on prior knowledge group, such that participants with high prior knowledge will have significantly higher proportions of fixations on AOIs related to learning and SRL and significantly lower proportions of fixations on the pedagogical agents than participants with low prior knowledge. Additionally, participants with high prior knowledge will engage in significantly higher frequencies of fixations between the text content and other learning and SRL-related AOI pairs, and significantly lower frequencies of fixations between the text content and pedagogical agent AOI pair than participants with low prior knowledge.

• (H3) There will be unique sequences of engaging in cognitive and metacognitive SRL processes for prior knowledge groups, such that participants with high prior knowledge will engage in sequences containing both cognitive and metacognitive processes, and participants with low prior knowledge will engage in sequences containing cognitive or metacognitive processes only; and

• (H4) There will be significant differences in sequences of cognitive and metacognitive SRL processes by prior knowledge group, such that while controlling for condition, participants with high prior knowledge will have significantly higher frequencies of sequences containing both cognitive and metacognitive SRL processes and significantly lower frequencies of sequences containing only cognitive or metacognitive SRL processes than participants with low prior knowledge.

Methods

Participants and Materials

The participants in this study were 194 undergraduate students from 3 large North American universities (N = 194 for the first analysis (53% female) and a subset of n = 30 for the eye tracking and sequence mining analyses (67% female)). Participants’ ages ranged from 18 to 41 years old (M = 20.47, SD = 2.92), and they were compensated $10 per hour, up to $40, for participating in the 2-day study.

We administered several self-report questionnaires on days 1 and 2 of the study. On day 1, we administered a consent form, demographics questionnaire, as well as a series of self-report measures on participants’ emotions (Achievement Emotions Questionnaire, part 1 [Pekrun et al. 2011], Emotion Regulation Questionnaire [Gross and John 2003]) and motivation (Achievement Goals Questionnaire [Elliot and Murayama 2008], Global Self-Esteem [Rosenberg et al. 1989; Rosenberg et al. 1995]). On day 2, we administered additional questionnaires about emotions towards the pedagogical agents of MetaTutor (Agent Persona Inventory [Baylor and Ryu 2003], Achievement Emotion Questionnaire, part 2 [Pekrun et al. 2011]) and motivation (Attributions for Post-Test Performance). However, for this study, we did not include any self-report measures as our goal was to investigate student behaviors from online trace data (log files, eye tracking), and not self-report measures.

We also administered a pre-test at the end of day 1, and a post-test at the end of day 2. The pre- and post-tests were 30-item counterbalanced, 4-choice multiple-choice questions about the circulatory system. Pre- and post-tests were developed by a graduate student in Biology with high expertise of the human circulatory system. Pre-test scores ranged from 7 (23%) to 28 (93%) out of 30 (M = 17.27, SD = 4.45), and post-test scores ranged from 5 (17%) to 29 (97%) out of 30 (M = 20.59, SD = 4.21). For this study, we used pre-test score to measure prior knowledge, and pre- and post-test scores to calculate proportional learning gain (see Coding and Scoring, below).

MetaTutor Study

During learning with MetaTutor, we collected a series of multichannel data including log files, eye tracking, videos of facial expressions, and electrodermal activity [EDA] (physiological data). Log files collected all time stamped activity (at the millisecond level) during learning, including mouse clicks on different locations on the interface and making metacognitive judgments, and keyboard input, including typing notes and setting sub-goals. Eye-tracking data captured participants’ fixations on and saccades between various areas of interest (AOIs) on the MetaTutor interface, where a fixation was defined as looking at an AOI for at least 80 ms (Salvucci and Goldberg 2000) and a saccade was defined as movements between fixations. Video data of facial expressions captured participants’ emotions during learning, and EDA measured participants’ physiological arousal levels during learning. For this study, we used log files and eye-tracking data only (see Coding and Scoring below).

Experimental Conditions

MetaTutor had (1) an experimental, prompt and feedback condition, and (2) a control condition, and participants were randomly assigned to one condition prior to learning. In the prompt and feedback condition (n = 95), participants were provided with prompts and feedback from the pedagogical agents. For example, a participant was prompted to make a summary by Sam the Strategizer, and once they submitted their summary, Sam provided feedback on whether the summary was correct (e.g., had the appropriate key words, was appropriate in length). In addition, when participants self-initiated the use of SRL processes by clicking the cognitive or metacognitive process on the SRL palette, they received feedback on this as well. Prompts and feedback were given verbally from the agents; however participants could also open the interaction log at the bottom of the interface (which they were shown during the introductory video) if they wanted to view this interaction (e.g., to view their quiz score). Additionally, participants in this condition were free to navigate through the content in any order they chose. Therefore, participants in this condition were encouraged to self-regulate their learning, but with the assistance of the pedagogical agents who acted as externally-regulating agents.

In the control condition (n = 99), participants were also encouraged to self-regulate their learning; however they were not provided any external regulation from the pedagogical agents. Therefore, participants were not provided with prompts from the agents to engage in cognitive and metacognitive SRL processes, nor were they provided with feedback when they did so on their own. For example, participants were not prompted to make a summary; however they could self-initiate making a summary by clicking on the SRL palette, but were not provided with any feedback on whether their summary contained the necessary key words and was appropriate in length. Although the agents did not provide feedback, they still spoke to the participants when they self-initiated the use of SRL processes, and Gavin still spoke to participants to administer self-reports. Thus, the agents did not provide any feedback, but they still spoke with the participants. Therefore in this condition, the pedagogical agents were still present on the screen; however their role was much more minimal. Participants in this condition could also navigate through the pages in the order they chose, and all content was the same for both conditions, and so the only true difference between the conditions were the roles of the pedagogical agents.

Experimental Procedure

MetaTutor was a 2-day study (see Fig. 2 for a breakdown of the study by day), which lasted approximately 3 h (for both days). On day 1, participants first signed a consent form followed by a demographics form, and then completed the self-report questionnaires (see Participants and Materials section, above). They were then presented with the 30-item pre-test on the circulatory system. The day 1 session lasted 30–60 min, and when participants were done (after completing the pre-test), they were paid for their participation.

Fig. 2

MetaTutor study timeline for day 1 and day 2

On day 2, participants returned to the lab where they sat 70 cm from the monitor (47 × 30 cm in size, with a screen resolution of 1680 × 1050 pixels), and were instrumented by calibrating their eye tracking and collecting a baseline for their facial expressions and physiological data. First, they put the EDA bracelet on their non-dominant hand (so as not to interfere with using the mouse) and were asked to sit still and relax for 6 min (to collect a baseline for their EDA and face video). Then, participants’ eye tracking was calibrated for the SMI RED 250 eye tracker (SMI, 2014) using a 9-point calibration, where the calibration was acceptable if their visual angle did not deviate more than 0.5° horizontally or vertically (i.e., 2 calibration scores were given, and both scores had to be less than 0.5). Once their eye tracking was calibrated, the EDA and video began recording using iMotions Attention Tool version 6.2 (iMotions 2016). Once the calibration was completed, participants began using MetaTutor. They were first presented with introductory videos to help them navigate through the system and set sub-goals, followed by the sub-goal setting phase (part of planning, an important part of SRL) where they set two sub-goals. Next, they began the 90-min learning session (after the sub-goal setting phase to account for varying times spent setting sub-goals), where they could select the text content and diagrams they wanted to view, and could engage in SRL processes. After the 90 min, participants completed the 30-item post-test, followed by questionnaires (see Participants and Materials section, and Fig. 2). Once completed, participants were debriefed, paid (day 2 lasted between 2 and 3 h including calibration, the learning session, and post-test questionnaires), and thanked for participating.

Coding and Scoring

For this study, we investigated if and how proportional learning gain, proportion of fixations, frequency of AOI pairs, and how sequential patterns differed by their prior knowledge group. To do so, we extracted data from the log files and eye tracking, and coded and scored the data to obtain our independent and dependent variables.

Prior Knowledge Groups

As our independent (or grouping) variable, we differentiated between participants with high vs. low prior knowledge, based on their pre-test scores (calculated post-hoc). We performed a median split on pre-test score out of 30; median = 17. We then assigned participants with pre-test scores above 17 to the high prior knowledge (HPK) group (n = 89; where n = 44 for the experimental condition, and n = 45 for the control condition) and participants with pre-test scores of 17 or lower to the low prior knowledge (LPK) group (n = 105; where n = 51 for the prompt and feedback condition, and n = 54 for the control condition). We used prior knowledge group as our independent variable with two levels for all subsequent analyses.

Proportional Learning Gain

A proportional learning gain score allowed us to account for the amount of points earned from pre-test to post-test, while considering what participants’ pre- and post-test scores actually were. To calculate proportional learning gain, we used the following formula (Witherspoon et al. 2008): (PostRatio - PreRatio) ÷ (1 - PreRatio), where PostRatio was post-test score out of 30 and PreRatio was pre-test score out of 30.

Eye-Tracking Data

We used eye-tracking data to calculate the proportions of fixations on nine AOIs, which were all different MetaTutor interface elements (described in the MetaTutor section). We used the Dispersion-Threshold Identification algorithm (Salvucci and Goldberg 2000) to capture data classified as either a fixation or saccade. A fixation was defined as focusing on one AOI for at least 80 ms with a dispersion not exceeding 100 pixels and a saccade was defined as rapid eye movements that occurred between fixations with a dispersion of more than 100 pixels. For this analysis, we used fixation data only, as we wanted to determine time participants spent focusing on interface elements, and not time between those fixations. Additionally, we coded participants’ fixations for their first completed sub-goal only because the eye tracker only collected data for 2 h, and many participants took longer than 2 h to complete two sub-goals (the timer paused when engaging in SRL processes). Therefore, using one sub-goal allowed us to include more participants with complete datasets.

We used Experiment Center 3.4 (SMI, 2014) to mark our nine AOIs, which were created based on MetaTutor’s interface elements (see Fig. 3): (1) timer, (2) table of contents, (3) text content, (4) diagram, (5) learning goal, (6) sub-goals, (7) pedagogical agent, (8) SRL palette, and (9) notes. Different MetaTutor layouts could be open at different times (e.g., normal layout with eight AOIs on the screen; full layout with the content, diagram, and agent AOIs only; and normal layout with notes open, as shown in Figs. 1 and 3, that contained all nine AOIs). We first coded which layout was present on screen at all times during the session by coding the screen recordings. This was an important step because some AOIs were only on screen for some layouts, and Experiment Center would detect the AOI as being there at all times if we did not make this distinction. For example, the notes were only open when the participant clicked ‘take notes’ on the SRL palette; however if Experiment Center was not given this information, there could be a fixation on the notes even when they were not there; i.e., if a quiz was being taken. Therefore, coding when each AOI was on screen was necessary to calculate accurate fixations on these AOIs. Once all AOIs were drawn and we coded time spent on each layout, we generated a data file for each participant, which included the start and end times on each AOI, allowing us to then compute frequencies and mean durations on each AOI. Since participants largely varied in the amount of time they completed one sub-goal (M = 77.057 min, range: 37.15 to 121.38 min), we calculated the proportion of time spent fixating on each AOI to account for this variance. These large fluctuations were due to the amount of time engaging in SRL processes, where participants who used more processes spent more time engaging in these processes, and thus had longer fixations. Engaging in SRL processes caused the timer to stop (i.e., the timer stopped if participants were prompted to engage in SRL processes, or if they clicked on the SRL palette), resulting in longer session times (with the same 90 min on the timer counting down). As such, we calculated the proportion of fixations for each AOI with the formula: [(Mean fixation duration) × (Frequency)]÷(Total session time), which we used to create nine proportion scores, one for each AOI.

Fig. 3

Screenshot of MetaTutor with Areas of Interest (AOIs). Note. ToC = table of contents

We also used the outputted data files to compute frequencies for pairs of AOIs, which contained the text content AOI, and was matched with each of the other eight AOIs, resulting in eight AOI pairs. AOI pairs were classified as being immediately following each other, where the AOI pair began with the text content. Using AOI pairs allowed us to examine evidence of planning, monitoring, and strategizing SRL-related behaviors. For example planning could be demonstrated by fixating on the content and learning goal to plan what needed to be read to accomplish the goal, or fixating on the content and table of contents to select which pages to read. Monitoring could be demonstrated by fixating on the content and timer to assess how much time they had remaining, or by fixating on the content and sub-goal progress bar to monitor if they had read enough to complete the sub-goal. Strategizing could be evidenced by fixating on the content and SRL palette as an indicator of SRL strategy deployment, by fixating on the content and notes as a cognitive learning strategy, by fixating on the content and diagram as evidence of coordinating informational sources, or by fixating on the content and agent as evidence of help-seeking behavior.

We chose AOI pairs because we analyzed eye-tracking data from a small sample size of participants’ first set sub-goal, which resulted in analyzing only 56% of the learning session. Furthermore, combining all possible pairs of AOIs, generated 324 pairs, and so we chose to focus on AOI pairs that began with text content only because the text was the foundation for learning about complex science topics, and to trigger the use of planning, monitoring, and strategizing, which usually involved reading first. Therefore, using the eye-tracking data, we computed proportions of fixations on nine AOIs, as well as eight AOI pairs, which we used as dependent variables for our analyses.

Sequence Mining

We used log files to investigate the sequences of engaging in cognitive and metacognitive processes during learning. Sequence mining allowed us to examine patterns of behaviors in an event, including how often these patterns occurred. For this study, we examined the event of using cognitive and metacognitive SRL processes, as evidenced from the log files (i.e., participants clicking on the SRL palette), and what patterns of using these processes looked like for participants with high vs. low prior knowledge. Each specific behavior was a self-initiated instance of using a cognitive or metacognitive process, and we coded each behavior separately. Specifically, we created separate codes (see Table 1) of behaviors, which contained two separate elements, with four codes in total (2 × 2). These two dimensions were: (1) type of user-initiated process (cognitive or metacognitive) and (2) level of accuracy (high or low). Cognitive processes were taking notes and making summaries, as they were types of learning strategies participants could use to help them understand the content, and metacognitive processes were judgments of learning, feelings of knowing, content evaluations, and monitoring progress towards goals, as these were all strategies participants could use to assess how well they were progressing by learning the content and spending their time appropriately to do so. Level of accuracy was defined differently for each strategy. Notes were run through Latent Semantic Analysis using an LSA Primer developed at CU Boulder (The Science and Applications of Latent Semantic Analysis Group), where we compared the text from each content page to the notes participants took on that page. A higher score indicated the notes and content were of similar content (i.e., the notes were more accurate), and so scores with 50% similarity and higher were coded as accurate, and scores lower than 50% were coded as inaccurate. Summaries were coded directly in MetaTutor where a summary was scored as accurate if it contained a set of key words and was appropriate in length (3 sentences). We coded summaries as being accurate if they were correct on their first attempt (deemed appropriate by the system’s rules), and inaccurate if they were not correct on their first attempt. For judgments of learning and feelings of knowing, participants had to rate on a 6-point scale the degree to which they felt they understood the content (judgment of learning) or the degree of familiarity they felt with the content (feeling of knowing). After making a selection, they were directed to a 3-item multiple-choice page quiz. We developed a coding scheme based on participants’ response to the scale and their results on the quiz, such that accuracy was coded based on their performance on the quiz in relation to their judgment. For example, if they responded that they felt they did not understand the material at all, a score of 0% would be an accurate judgment, while a response of “I feel I definitely know the content” and a score of 100% would be accurate. A response of 1 out of 6 was matched with a quiz score of 0% for accuracy, 2 out of 6 with 0% or 33% (the response on the scale indicated less confidence, but not a clear response), 3 out of 6 with 33%, 4 out of 6 with 66%, 5 out of 6 with 66% or 100%, and 6 out of 6 with 100%. For content evaluation, an accurate score was given when the participant correctly identified the relevancy of the text and diagram, while an inaccurate score was given when they were not correctly identified. Finally, for monitoring progress towards goals, which required participants to respond to a 10-item multiple choice quiz, an accurate response included scores of 60% or higher (as identified by the system), while an inaccurate response included scores lower than 60%. Based on the type of process and level of accuracy for each specific strategy, we assigned 1 of 4 codes to each instance.

Table 1 Codes used for sequential pattern mining

We used these four codes to examine if there were unique patterns of engaging in these processes during learning for each prior knowledge group. We examined user-initiated SRL processes only because we were interested in examining SRL behavior, and not behavior related to external regulation from the pedagogical agents. As such, we were not required to distinguish between conditions because self-initiated behaviors were identical in both conditions. In addition, based on our analyses, there was no effect of condition for our dependent variables. Once we coded each instance of a cognitive or metacognitive behavior, our data were ready to be run through sequence mining algorithms. Overall, the abovementioned coding and scoring prepared us for conducting data analyses and obtaining results regarding the differences in students’ SRL behavior by their prior knowledge groups.

Results

For our analyses we used prior knowledge group as our independent variable (see above) and condition as our covariate to allow us to control for any effect of condition on our results. We ran factorial ANOVAs to examine the interaction between prior knowledge group and condition on our dependent variables and did not find any significant interactions (p > .05), confirming that there was no effect of condition on prior knowledge group, and the students  were split appropriately by condition.

Research Question 1: Are there Significant Differences in Proportional Learning Gain by Prior Knowledge Group, while Controlling for Condition?

We performed an ANCOVA with prior knowledge group as our independent variable (2 levels), proportional learning gain as our dependent variable, and condition as our covariate. Results revealed (see Table 2) a significant effect; F(1,191) = 7.22, p < .01, η_p² = .036. This suggests that while controlling for condition, there was a significant difference in proportional learning gain between prior knowledge groups, such that participants with high prior knowledge had significantly lower proportional learning gains (M = .16, SD = .42) than participants with low prior knowledge (M = .29, SD = .22). A proportional learning gain of .16 signifies an increase in 16% from pre-test score to post-test score, where a proportional learning gain of .29 signifies a gain in 29% from pre-test score to post-test score. Therefore, these results reveal that participants with high prior knowledge gained, on average, 16% (or 5 points) from pre-test to post-test, which was significantly fewer points than participants with low prior knowledge who, on average, gained 29% (or 9 points) form pre-test to post-test.

Table 2 Descriptive statistics and results for proportional learning gain by PK group (n = 194)

Research Question 2: Are there Significant Differences in Fixation Behavior by Prior Knowledge Group, while Controlling for Condition?

For the following research questions, we used a subset of the data, which included participants with full eye-tracking datasets (n = 30). Participants remained in the same prior knowledge group as originally assigned. Therefore, from this dataset, there were 19 participants in the HPK group (with 11 participants in the prompt and feedback condition, and 8 in the control condition) and 11 participants in the LPK group (with 6 in the prompt and feedback condition, and 5 in the control). We performed two MANCOVAs and one ANCOVA with prior knowledge group as our independent variable (2 levels), proportion of time spent fixating on AOIs as our dependent variables, and condition as our covariate. Our first MANCOVA investigated AOIs related to learning and knowledge acquisition, which contained four AOIs: text, diagram, notes, and table of contents. Results (see Table 3) did not reveal a significant multivariate effect; Wilks’ λ = .82, F(4, 24) = 1.31, p = .29, η_p² = .18, suggesting that there were no significant differences in the proportion of time spent fixating on the text, diagrams, notes, or table of contents between prior knowledge groups.

Table 3 Descriptive statistics and results for proportions of fixations on AOIs by PK group (n = 30)

Our second MANCOVA used AOIs related to monitoring SRL processes: timer, SRL palette, sub-goals, and overall learning goal, as our dependent variables. Results (see Table 3) were not significant; Wilks’ λ = .84, F(4, 24) = 1.17, p = .35, η_p² = .16, suggesting that there were no significant differences in the proportion of time spent fixating on the timer, SRL palette, sub-goals, or overall learning goal between prior knowledge groups. An ANCOVA (with the agent AOI as a DV) was not significant; F(1, 27) = .23, p = .64, η_p² = .008 (see Table 3), revealing no significant differences in the proportion of time spent fixating on the agents between prior knowledge groups.

Overall, these results reveal that there were no significant differences in the proportion of time fixating on all AOIs during learning with MetaTutor between prior knowledge groups. However, we also performed several chi-square analyses on the frequencies of fixations from the content AOI to all other AOIs to investigate if participants with different levels of prior knowledge transitioned from the text content to different AOIs during learning. We performed eight chi-square analyses (α = .00625), which consisted of combinations of the text content with the other eight AOIs. Results (see Table 4) revealed significant effects for the text content-diagram AOI pair (χ²(1) = 119.16, p < .001), the text content-notes AOI pair (χ²(1) = 187.3, p < .001), the text content-table of contents AOI pair (χ²(1) = 185.49, p < .001), the text content-SRL palette AOI pair (χ²(1) = 14.77, p < .001), the text content-sub-goals AOI pair (χ²(1) = 11.76, p < .001), and the text content-learning goal AOI pair (χ²(1) = 15.52, p < .001). Specifically, participants with high prior knowledge had significantly higher frequencies of the text content-diagram AOI pair, the text content-notes AOI pair, the text content-table of contents AOI pair, the text content-SRL palette AOI pair, the text content-sub-goals AOI pair, and the text content-learning goal AOI pair than participants with low prior knowledge (see Table 4 with all frequencies). Therefore, although we did not find significant differences in fixations on individual AOIs between prior knowledge groups, we did find significant differences in the frequencies of fixations on AOI pairs between prior knowledge groups, demonstrating that participants with different levels of prior knowledge might have been engaging in different SRL processes during learning, as indicated by their frequencies of fixations from the text content to other AOIs during learning.

Table 4 Chi-square results for AOI pairs by PK group (n = 30)

Research Question 3: Can we Detect Sequences of Engaging in Cognitive and Metacognitive SRL Processes by Prior Knowledge Group?

We used sequential pattern mining to investigate if participants with different levels of prior knowledge engaged in different sequences of cognitive and metacognitive SRL processes during learning. We used the SPAM algorithm (Ayres et al. 2002; Fournier-Viger et al. 2014), which outputted different support values of sequences of user-initiated SRL processes. Support values are defined as the amount of participants a sequence occurred at least one time for (i.e., a support value of 100% would indicate the given sequence occurred at least once for each participant in that group). We ran the algorithm separately for each prior knowledge group, generating a list of all enacted sequences, along with their support values, for each prior knowledge group. We examined sequences with 2–3 length codes because we were limited to a smaller sample size (n = 30), as we wanted to use the same sample we used for the eye-tracking analyses, and using 2-length sequences was analogous to examining the AOI pairs, where examining 3-length sequences allowed us to examine if there were transitions back to the initial behavior (e.g., 2 → 5 → 2).

Results revealed that overall, we were able to extract 2- and 3-length sequences for each prior knowledge group (see Table 5). Specifically, results revealed that there were 20 sequences with a support value of at least 50% (the sequence appeared at least once for half of the participants) for high prior knowledge participants, and there were 10 sequences with a support value of 50% for low prior knowledge participants. Results also revealed that there was only 1 sequence with a support value over 80% for both prior knowledge groups (see Table 5). Both sequences contained only metacognitive codes (5 or 6), where 5 is an accurate metacognitive judgment (with a SV of 84% for high prior knowledge participants) and 6 is an inaccurate metacognitive judgment (with a SV of 82% for low prior knowledge participants). As such, it appears that a majority of participants with high prior knowledge were engaging in sequences with two accurate metacognitive judgments, where a majority of participants with low prior knowledge were engaging in sequences with inaccurate metacognitive judgments. Lastly, results revealed that participants did not frequently engage in sequences containing both cognitive and metacognitive processes, such that participants with high prior knowledge engaged in more cognitive than metacognitive processes, and some sequences containing both (although not statistically examined for this research question), while participants with low prior knowledge did not engage in any sequences containing cognitive processes (or both cognitive and metacognitive processes). Overall, these results suggest that we were able to detect sequences of engaging in cognitive and metacognitive SRL processes during learning with MetaTutor for participants with both levels of prior knowledge.

Table 5 Sequences of 2–3 length codes by PK group

Research Question 4: Are there Significant Differences in Sequences of Cognitive and Metacognitive SRL Processes by Prior Knowledge Group, while Controlling for Condition?

We used a differential sequence mining approach (Kinnebrew et al. 2013), where we compared the instance support values of a selected set of sequences between prior knowledge groups. Instance support values are the frequencies of occurrence of each sequence for each participant (i.e., how often a sequence occurred for each participant in that group). Thus, differential sequence mining allowed us to compare for significant differences in the frequency of occurrence of sequences between prior knowledge groups. For our analyses, we chose 2-length sequences because those sequences had the highest frequencies (i.e., the 3-length sequences had very low instance support values), and we wanted to include sequences with higher support and instance support values.

We ran a MANCOVA with prior knowledge group as the independent variable (2 levels), eight sequences as our dependent variables, and condition as our covariate. Results revealed a significant multivariate effect; Wilks’ λ = .49, F(8, 20) = 2.65, p = .037, η_p² = .52. Subsequent univariate analyses (see Table 6) revealed a significant effect for sequence 1 ➔ 1 [CogHu ➔ CogHu]); F(1, 27) = 8.15, p < .01, η_p² = .23, sequence 6 ➔ 6 [MetacogLu ➔ MetacogLu]; F(1, 27) = 5.49, p = .027, η_p² = .17, sequence 1 ➔ 5 [CogHu ➔ MetacogHu]; F(1, 27) = 7.11, p = .013, η_p² = .21; sequence 5 ➔ 1 [MetacogHu ➔ CogHu]; F(1, 27) = 4.34, p = .047, η_p² = .14; and sequence 1 ➔ 2 [CogHu ➔ CogLu]; F(1, 27) = 5.61, p = .025, η_p² = .17. Specifically, while controlling for condition, participants with high prior knowledge had significantly higher frequencies of sequences 1 ➔ 1 [CogHu ➔ CogHu], 1 ➔ 5 [CogHu ➔ MetacogHu], 5 ➔ 1 [MetacogHu ➔ CogHu], and 1 ➔ 2 [CogHu ➔ CogLu], and significantly lower frequencies of sequence 6 ➔ 6 [MetacogLu ➔ MetacogLu] than participants with low prior knowledge.

Table 6 Univariate Results for Instance Support Values by PK Group (n = 30)

Discussion

The goal of this study was to use log-file and eye-tracking data to investigate how different levels of prior knowledge impacted participants’ proportional learning gain, proportions of fixations on areas of interest related to engaging in SRL, frequencies of fixations on pairs of areas of interest indicative of engaging in SRL, and sequences of engaging in cognitive and metacognitive SRL processes, all during learning with MetaTutor, an ITS that fosters the use of SRL processes during learning about the circulatory system.

Results from research question 1 revealed that participants with low prior knowledge had significantly higher proportional learning gains than participants with high prior knowledge, where 3.6% of the variance in proportional learning gain was accounted for by their prior knowledge group. This does not support H1, as we predicted the opposite effect. However, given that the low prior knowledge group had lower pre-test scores to begin with, these results demonstrate that these participants simply had more points to gain. Therefore, participants with high prior knowledge did, on average, increase from pre- to post-test; however they did not gain as much as participants with low prior knowledge. These results align with the IPT model (Winne and Hadwin 1998, 2008; Winne 2018) because we did find differences based on the cognitive condition of prior knowledge, as posited in the model. Additionally, these results do not align with Greene et al. (2010) who found a significant positive association between prior knowledge and performance, perhaps because they examined performance, and not proportional learning gain, therefore not taking into account the amount of points gained, and just assessing who had higher scores. These results demonstrate the differences between examining the impact of prior knowledge on overall performance and proportional learning gain.

Results from our second research question revealed that there were no significant differences between students’ prior knowledge groups for the proportions of fixations on areas of interest related to learning (text content, diagram, notes, table of contents), nor were there significant differences between prior knowledge groups in the proportions of fixations on areas of interest related to SRL processes (timer, sub-goal, learning goal, SRL palette) or the area of interest of the pedagogical agent. This did not align with H2 where we expected significant differences in proportions of fixations on all of these nine AOIs by students’ prior knowledge group. Based on these results, it is possible that investigating time of fixations on a single AOI was too micro to reveal how participants engaged in SRL processes, and examining transitions between AOIs might be more indicative of engaging in SRL processes. Therefore, as a second part of this research question, we investigated the frequencies of fixations on AOI pairs (from text content to each of the other eight AOIs), and examining AOI pairs allowed us to detect these significant differences in fixations between prior knowledge groups. Results revealed that participants with high prior knowledge had significantly higher frequencies in the text content-diagram, text content-notes, text content-table of contents, text content-SRL palette, text content-sub-goals, and text content-learning goal AOI pairs than participants with low prior knowledge, while there were no significant differences between prior knowledge groups in the frequencies of the text content-timer or text content-agent AOI pairs. These results partially support our hypotheses because we expected participants with high prior knowledge to engage in higher frequencies of most AOI pairs except the text content-agent AOI pair than low prior knowledge participants; however, we did not find any differences in the text content-timer AOI pair. In addition, we also hypothesized that the low prior knowledge group would have higher frequencies of the text content-agent AOI pair than the high prior knowledge group, which we did not find, not supporting H2. Therefore, it appears that high prior knowledge participants spent more time fixating on AOIs related to knowledge acquisition and monitoring, perhaps because they did not need to spend extra time learning the content since they had high prior knowledge and were able to transition more often from the text content to the other AOIs, as opposed to participants with low prior knowledge who did not have the opportunity to also fixate on other AOIs on the interface because they had to spend mote time reading the content. The non-significant differences reveal that all participants might not have been monitoring their time fixating from the text content to the timer, and participants were not turning to the agent for help seeking by fixating from the text content to the agent, given the low frequencies of fixations on both of these AOI pairs compared to the others.

These results support the IPT model of SRL (Winne and Hadwin 1998, 2008; Winne 2018) because we are again finding the impact of cognitive conditions (via prior knowledge) on SRL; however these results also lead us to reconsider the definition of an event, as defined by the model. Specifically, the IPT model posits that SRL is an event; however when defining an event using different types of multichannel data, we might want to define an event as a series of activities or behaviors contributing to that one event. For example, if we are investigating SRL with eye-tracking data, examining a transition (or sequence of transitions) from one AOI to another, as evidenced by investigating AOI pairs in this study, can be more indicative of SRL instead of examining fixations on a single AOI separately. Furthermore, these results align with Moos and Azevedo (2008) who found positive associations between prior knowledge with planning and monitoring, while we found higher frequencies of these processes for high prior knowledge participants. There was a negative association between strategizing and prior knowledge; however we found high prior knowledge participants used more cognitive strategies as well. Our results also did not align with Trevors et al. (2014), who found more notes for participants with low prior knowledge (in the prompt and feedback condition). Therefore, these results might reveal the differences between examining transitions (via eye tracking) and single events (via single clicks or think-aloud protocols).

Results from our third research question revealed that we were able to detect sequences of engaging in cognitive and metacognitive SRL processes for each prior knowledge group. Additionally, participants with high prior knowledge had sequences containing both cognitive and metacognitive processes, while participants with low prior knowledge did not. Therefore, H3 was partially supported because we hypothesized that we would find unique sequences, which we did. We also hypothesized that high prior knowledge participants would exhibit sequences containing both cognitive and metacognitive processes; however both prior knowledge groups had sequences with only metacognitive processes, and low prior knowledge participants did not have any sequences with cognitive processes only for at least 50% of participants, thus partially supporting H3. These results suggest that participants with low prior knowledge might not have had enough cognitive resources (i.e., high cognitive load; Moos 2013; Paas et al. 2004) to engage in both cognitive and metacognitive processes because they had to allocate some of their resources to learning the material, while participants with high prior knowledge did not. Specifically, within the MetaTutor environment, it can be beneficial to first engage in a metacognitive strategy, followed by a cognitive one. For example, a participant should first assess if they understand the material before they take notes on it, or they should first evaluate the relevancy of the page to their current sub-goal before choosing to read the page or selecting a different page to read. Therefore, our results suggest that participants with low prior knowledge were able to make the metacognitive judgment, but since they had to learn the material as well, they did not follow the metacognitive judgment with engaging in subsequent cognitive processes, such as taking notes. We also observed this in the previous research question, where participants with high prior knowledge fixated more from the text to the diagram, their notes, the SRL palette, their sub-goals, and their learning goal than participants with low prior knowledge, who had lower frequencies of these AOI pairs We also found sequences with only cognitive or metacognitive processes per sequence, which might be indicative of trying to perform better than they had on their first attempt, but this is an area for future direction, as we cannot conclude this without knowing if processes occurred on the same page.

These results align with the IPT model (Winne and Hadwin 1998, 2008; Winne 2018) because we did discover that all participants are using some types of SRL processes, indicating they are engaging in phases of SRL, and again that the cognitive condition of prior knowledge continues to impact the use of SRL processes. The model does not specify the order in which to use cognitive and metacognitive processes, which solely relies on the specific ITS used. Therefore, we do not interpret the specific order of codes within the sequences based on the IPT model, but rather on MetaTutor itself. Moreover, similar to our previous research question, these results do not align with Moos and Azevedo (2008) or Trevors et al. (2014) because they found a negative association between prior knowledge and strategizing and that low prior knowledge participants took more notes, respectively, and we found that only participants with high prior knowledge had sequences using cognitive strategies. However, Moos & Azevedo found a positive relationship between prior knowledge and planning and monitoring, and we did see high prior knowledge participants engage in sequences with metacognitive monitoring processes. Finally, our results did not align with Taub et al. (2014), who found high prior knowledge participants engaged in more metacognitive rather than cognitive processes, which was the opposite of our findings, and demonstrates how different datasets can reveal quite different results, as these studies used different versions of MetaTutor, and Taub et al. found a higher median than the current sample (20 vs. 17), nor did they use eye-tracking data.

Results from our fourth research question revealed that we detected significant differences in the frequencies of occurrence (i.e., instance support values) of sequences containing cognitive and metacognitive SRL processes. Results partially supported H4, as we hypothesized that high prior knowledge participants would have more sequences containing both cognitive and metacognitive strategies. We did find that high prior knowledge participants had significantly higher frequencies of sequences 5 ➔ 1 [MetacogHu ➔ CogHu] (where 14% of the variance in instance support value for this sequence was accounted for by prior knowledge group) and 1 ➔ 5;CogHu ➔ MetacogHu] (where 21% of the variance in instance support value for this sequence was accounted for by prior knowledge group) than low prior knowledge participants. However they also had higher frequencies of the 1 ➔ 1 [CogHu ➔ CogHu] (where 23% of the variance in instance support value for this sequence was accounted for by prior knowledge group) and 1 ➔ 2 [CogHu ➔ CogLu] (where 17% of the variance in instance support value for this sequence was accounted for by prior knowledge group) sequences than low prior knowledge participants, which we did not hypothesize. Participants with low prior knowledge had significantly higher frequencies of the 6 ➔ 6 [MetacogLu ➔ MetacogLu] (where 17% of the variance in instance support value for this sequence was accounted for by prior knowledge group) sequence, and we did hypothesize that participants with low prior knowledge would have significantly higher frequencies of sequences containing only cognitive or metacognitive processes. The 6 ➔ 6 sequence contains two codes of low metacognitive processes, which indicates that they performed poorly on the first attempt and tried again and performed poorly again, or they engaged in consecutive metacognitive processes poorly on different pages. Participants with high prior knowledge engaged in high accuracy cognitive and metacognitive processes (codes 1 and 5), but they also engaged in sequence 1 ➔ 2 (high cognitive-low cognitive), which needs further investigation to examine if this was within one instance of a cognitive event, or two separate events. We would expect the 5 ➔ 1 [MetacogHu ➔ CogHu] code because that indicates a metacognitive and then a cognitive process, which we indicated was an efficient sequence to deploy during learning with MetaTutor. However we also found significantly higher frequencies of 1 ➔ 5 [CogHu ➔ MetacogHu] for high prior knowledge participants, which means these events might have occurred on different pages, or that sometimes participants with high prior knowledge take notes or summarize first, and then make metacognitive judgments. Finally, our results did not reveal any significant differences in sequences containing only metacognitive processes (besides 6 ➔ 6 [MetacogLu ➔ MetacogLu], and we did not expect high prior knowledge participants to engage in poor metacognitive strategies), which is supported using our interpretation regarding cognitive load (Moos 2013; Paas et al. 2004), such that we do not see differences in metacognitive processes because all participants have the resources to engage in them, but only participants with high prior knowledge have the cognitive capacity to engage in cognitive processes as well.

These results align with the IPT model (Winne and Hadwin 1998, 2008; Winne 2018), as we see participants are progressing through the phases of SRL, which is impacted by the cognitive condition: prior knowledge. Again, our results do not align with Trevors et al. (2014) as we do not see the use of cognitive processes for participants with low prior knowledge, and they found a significant association between low prior knowledge and note-taking (a cognitive process). Our results do not align with Taub et al. (2014) who found differences in metacognitive, but not cognitive strategies between prior knowledge groups. Finally, we are seeing a positive effect of prior knowledge on the use of metacognitive monitoring strategies, which does align with Moos and Azevedo (2008), however we did not find the negative effect of strategizing, but instead found a positive relationship with prior knowledge and strategizing. Therefore, these results reveal that investigating sequences of engaging in cognitive and metacognitive SRL processes aligns with results from our previous research questions, revealing that we can use different types of data channels to investigate how participants with different levels of prior knowledge used SRL processes during learning with MetaTutor.

As such, although our findings were not all significant, the results from this study allowed us to demonstrate that we can use multichannel process data to assess how students engage in SRL, where SRL is a series of events that unfold over time (Winne and Hadwin 1998, 2008; Winne 2018). By identifying patterns of SRL-related behavior for participants with different levels of prior knowledge, this allows us to gain a deeper understanding of how SRL differs for students of varying ability levels, and how we can provide them with additional scaffolding that will cater to these differences. For example, if students with low prior knowledge are failing to use cognitive strategies, the system can prompt the use of these strategies during learning. In contrast, the system will not have to prompt the use of these strategies for students with high prior knowledge, therefore eliminating unnecessary prompting, which might have lead students with high prior knowledge to become frustrated from prompts that do not apply to them. Therefore, it is important to identify students’ learning characteristics, such as prior knowledge, before they learn with these systems, as it can allow for efficient learning for everyone.

Limitations

Although our results examined SRL as an event using process data, we must acknowledge the limitations. First, we do not have validity and reliability measures of our pre- and post-tests. Next, although we did differentiate between cognitive and metacognitive processes, we only coded for two cognitive processes, while we coded for four metacognitive processes. Therefore, just because participants with low prior knowledge did not reveal sequences engaging in cognitive processes, this only means they did not take as many notes or summarize, not that they did not attempt to use other cognitive learning strategies, such as coordinate information from the text with information from the diagram (coordination of informational sources). Furthermore, we coded all processes differently based on the nature of the process, and although we ended up with accurate or inaccurate ratings, it is possible that if we included weighted scores, results might have been different. Future studies should, therefore, aim to include codes with more weighted scores. Finally, we did not include contextual information in our codes, such as page numbers and durations of processes, therefore not allowing us to determine if codes occurred within the same page, and therefore within the same SRL event (e.g., engage in a second attempt to improve the accuracy of a summary), or if they were unique events (e.g., taking a page quiz on separate pages). Therefore, future analyses should include this information so we can even better understand not only which SRL processes participants engaged in and how accurate they were, but also when did participants use them and why. Finally, our sample size dropped to only 30 participants for the eye-tracking analyses and sequence mining, and so we should aim to include a larger sample size in future analyses. These limitations pave the way for engaging in future studies to even further investigate SRL processes during learning with ITSs.

Future Directions

There are many directions we can take to further investigate the impact of prior knowledge on SRL, including using different types of multichannel data, such as videos of facial expressions of emotions, and employing more advanced statistical techniques, such as using educational data mining techniques on eye-tracking data (e.g., see Azevedo et al. 2018).

For this study, we used eye-tracking data to investigate proportions of fixations on single AOIs, and fixations on AOI pairs using traditional statistics (MANOVAs, chi-squares). Although the chi-squares did generate statistically significant results, we were limited to selecting AOI pairs that began with text content (to avoid running over 50 chi-squares). Future studies should seek to investigate sequences of fixations by students’ prior knowledge group, which can include sequence lengths longer than 2 AOIs, as well as sequences with combinations of all AOIs that are not limited to starting with the same sequence. Perhaps by looking at sequences of AOIs, we can find the significant differences we were not able to observe with the MANCOVAs. Moreover, we chose to use MANCOVAs for our analyses to use consistent analyses across research questions; however in future studies, we will aim to use different approaches for comparing sequential patterns between groups, such as matching techniques.

In addition, future research can aim to include more data channels, such as videos of facial expressions of emotions, and we can also include our self-report data of emotions and motivation to see if there are differences in participants’ levels of emotions and motivations between prior knowledge groups, and how this impacts their SRL. Specifically, future studies can aim to investigate how emotions impact participants with different levels of prior knowledge, as we have seen in previous research that emotions do play an important role in learning (D’Mello and Graesser 2012). For example, research investigating action units (AUs) has revealed that furrowing of the eyebrows (AU4) can be indicative of both confusion and/or mental effort. Therefore, can this distinction be made based on prior knowledge, such that participants with low prior knowledge with high levels of AU4 are expressing confusion because of their low prior knowledge, which results in engaging in less accurate metacognitive monitoring processes, and not engaging in cognitive learning processes at all? In contrast, are participants with high prior knowledge not exhibiting confusion with high levels of AU4, but are actually exhibiting high levels of mental effort, resulting in accurate cognitive and metacognitive SRL processes? Future research should seek to investigate not only how prior knowledge impacts the use of cognitive and metacognitive SRL processes, but affective and motivational processes as well.

The ultimate goal of designing ITSs and examining their ability to enhance SRL and overall learning is to make these ITSs adaptive based on multichannel data, so that the system can provide individualized tutoring based on students’ individual learning needs (e.g., level of prior knowledge) and multichannel data to ensure that they are learning the complex topic, and are engaging in effective and efficient SRL processes. For example, if the system detects a student has low prior knowledge, the ITS can provide scaffolding that might guide them towards using both cognitive and metacognitive SRL processes during learning, similar to those sequences we detected in participants with high prior knowledge. As such, these results can be implemented towards developing ITSs that scaffold and adapt to learners based on these already-detected types of behaviors. In this study, we determined that we could extract sequences of how participants with high and low levels of prior knowledge engaged in cognitive and metacognitive SRL processes during learning with an ITS. We can apply these results towards further understanding student SRL during learning with ITSs. We can also use these results as progress towards developing student models within ITSs to make then more adaptive to individual students with different individual differences characteristics, such as prior knowledge, to ensure students with different ability levels can gain a positive experience during learning with these systems.