Mastery-Oriented Shared Student/System Control Over Problem Selection in a Linear Equation Tutor
- Cite this paper as:
- Long Y., Aleven V. (2016) Mastery-Oriented Shared Student/System Control Over Problem Selection in a Linear Equation Tutor. In: Micarelli A., Stamper J., Panourgia K. (eds) Intelligent Tutoring Systems. ITS 2016. Lecture Notes in Computer Science, vol 9684. Springer, Cham
Making effective problem selection decisions is a challenging Self-Regulated Learning skill. Students need to learn effective problem-selection strategies but also develop the motivation to use them. A mastery-approach orientation is generally associated with positive problem selection behaviors such as willingness to work on new materials. We conducted a classroom experiment with 200 6th – 8th graders to investigate the effectiveness of shared control over problem selection with mastery-oriented features (i.e., features that aim at fostering a mastery-approach orientation that simulates effective problem-selection behaviors) on students’ domain-level learning outcomes, problem-selection skills, enjoyment, future learning and future problem selection. The results show that shared control over problem selection accompanied by mastery-oriented features leads to significantly better learning outcomes, as compared to fully system-controlled problem selection, as well as better declarative knowledge of a key problem-selection strategy. Nevertheless, there was no effect on future problem selection and future learning. Our experiment contributes to prior literature by demonstrating that with tutor features to foster a mastery-approach orientation, shared control over problem selection can lead to significantly better learning outcomes than full system control.
KeywordsMastery-approach orientation Problem selection Self-Regulated Learning Learner control Classroom experiment Intelligent Tutoring System
Intelligent Tutoring Systems often are strongly system-controlled learning environments that adaptively select problems for students based on their knowledge level . Recently, some ITSs have started to grant students control to select their own learning tasks to elicit higher motivation, which in turn may lead to better learning outcomes . However, prior research has found that students are not good at making effective problem selection decisions . Fully student-controlled problem-selection was found to lead to worse learning outcomes than system-selected problems . Hence some ITSs created shared student/system control over problem selection (e.g., letting the system pick problem types while the students select a specific problem from that type) to prevent students from making suboptimal decisions and achieved comparable learning outcomes to those achieved with full system control . It is still an open question how ITSs can be designed to foster better learning outcomes and higher motivation with shared control over problem selection, as compared to full system control. In addition, theories of SRL emphasize the important role of motivation in promoting desirable SRL behaviors . Yet little work with ITSs has adopted a motivational design (i.e., design to foster motivations) approach to foster appropriate motivations that will stimulate effective problem-selection behaviors. Most of the interventions that support SRL processes in ITSs use cognitive and metacognitive tools, such as prompts and feedback [3, 11]. Furthermore, few of these studies have measured the lasting effects of the interventions when they are not in effect .
We tackle these open questions by applying motivational design to extend an ITS for equation solving, Lynnette, to help students learn an effective problem-selection rule, i.e., the Mastery Rule, while fostering a mastery-approach orientation . The Mastery Rule specifies that students should stop practicing a problem type once it is fully mastered. An ITS that implements this rule (in a system-controlled manner) led to better learning outcomes than a fixed curriculum . Our prior classroom studies and interviews with students revealed that the lack of a mastery-approach orientation might be a main challenge that keeps students from applying the Mastery Rule when they can select problems for themselves . A mastery-approach orientation is a type of achievement goal that is associated with positive learning behaviors such as perseverance and willingness to learn new materials . It aligns with the desirable problem selection behaviors based on the Mastery Rule. It is likely, but unproven that students with a mastery-approach orientation will apply the Mastery Rule to select problems and achieve better learning outcomes in the tutor, as compared to full system control. We therefore added features to foster a mastery-approach orientation. We refer to these features as the mastery-oriented features.
The current paper describes our classroom experiment that investigated two research questions: Research Question 1: Compared to full system control over problem selection, does shared control, supported by mastery-oriented features enhance students’ (a) problem-selection decisions in the tutor; (b) domain-level learning outcomes; (c) enjoyment and (d) knowledge of the Mastery Rule? Research Question 2: Do the mastery-oriented features enhance students’ (a) future problem-selection decisions and (b) future domain-level learning outcomes in an environment with shared control but without mastery-oriented features, as compared to full system control?
2.1 Experimental Design
Conditions of the learning phase and the future learning phase
2.2 Procedure, Measurements and Participants
Overview of the procedure and measurements of the experiment
Future learning phase
• 6 items on equation solving abilities of levels 1–6
• 4 41-min class periods
• Learning the first 6 levels
• Mid-Test-Equations1: 6 items on levels 1–6
• Mid-Test-Equations2: 3 items on levels 7–9
• 7 7-point Likert scale items on enjoyment of using the system
• 3 items on declarative knowledge of applying the Mastery Rule
• 2 41-min class periods
• All 9 levels were unlocked
• Post-Test-Equations1: 6 items on levels 1–6
• Post-Test-Equations2: 3 items on levels 7–9
200 students completed the pre-test and mid-test, and were present in all four class periods or mastered the first six levels during the Learning Phase. We refer to these 200 students as the Learning-Phase-Sample. 165 students completed the pre-test, mid-test and post-test. They were present during all 6 class periods (both the Learning and Future Learning Phases) or mastered all 9 levels. These students constitute the Future-Learning-Phase-Sample. We report Cohen’s d for effect sizes. An effect size d of .20 is typically deemed a small effect, .50 a medium effect, and .80 a large effect. For all ANCOVAs, Teacher was used as a co-variate to account for the variances that reside within different teachers’ classes.
3.1 The Learning Phase: Research Questions 1.a – 1.d
We first analyzed data from the Learning Phase, to answer Research Questions 1.a-1.d. The Learning-Phase-Sample was used for all analyses.
Problem Selection Decisions (RQ 1.a). To test the hypothesis that mastery-oriented features will help foster more consistent application of the Mastery Rule, we looked at the percentage of mastered problems the students selected in the “Mastery Shared” condition during the Learning Phase (under perfect application of the Mastery Rule, the students should not select any mastered problems). Twenty out of 102 students (19.61 %) in the “Mastery Shared” condition selected at least one mastered problem during the Learning Phase. On average 1.4 % of the problems (SD = 3.8 %) selected by each student in the condition were mastered problems, indicating good application of the Mastery Rule when the mastery-oriented features were present.
Means and SDs for test performance of levels 1–6 equations on pre-test and mid-test
Given the ceiling effect on the pre-test, we split the sample based on the median of the pre-test score (median = .83) into two sub-groups: the Lower-Performing Group and the Higher-Performing Group. The Lower-Performing Group had 102 students (mean pre-test = 0.67, SD = 0.18), and the Higher-Performing Group had 98 students (mean pre-test = 0.98, SD = 0.05). ANCOVAs revealed that overall the two conditions improved significantly from pre-test to mid-test on Equations1 within the Lower Performing Group (F (1, 94) = 13.451, p < .000, d = .76). The condition effect was marginally significant (F (1, 94) = 3.490, p = .065, d = .37), with the “Mastery Shared” condition improving more than the “Standard Tutor” condition. On the other hand, there was a significant decrement of the two conditions’ performance from the pre-test to mid-test within the Higher-Performing Group (F (1, 90) = 25.704, p < .000, d = 1.07), probably representing regression to the mean. No significant condition effect was found for the learning gains within the Higher-Performing Group.
Enjoyment (RQ 1.c). To test the hypothesis that mastery-oriented shared control over problem selection will lead to higher enjoyment of using the tutor than full system control, we compared students’ enjoyment ratings on the mid-test. The “Mastery Shared” condition reported higher enjoyment (mean = 4.63, SD = 1.59) than the “Standard Tutor” (mean = 4.52, SD = 1.36). However, an ANCOVA test found the difference was not statistically significant (F (1, 192) = .450, p = .530, d = .09).
Declarative Knowledge (RQ 1.d). To test the hypothesis that the mastery-oriented features with shared control will lead to better knowledge of the Mastery Rule, compared to full system control, we analyzed the students’ responses to the three items on the mid-test. There were 12 options for all three items. The students were instructed to check all options that apply. We coded the students’ responses to each option as 0 or 1. On average those in the “Mastery Shared” condition (mean = 0.76, SD = 0.15) scored significantly higher (F (1, 184) = 8.263, p = .005, d = .59) than those in the “Standard Tutor” condition (mean = 0.69, SD = 0.17). The “Mastery Shared” condition showed significantly better declarative knowledge of the Mastery Rule on the mid-test after the Learning Phase.
3.2 The Future Learning Phase: Research Questions 2.a and 2.B
We performed analyses on students’ problem selection decisions and equation solving performance. The Future-Learning-Phase-Sample was used for all analyses.
Problem Selection Decisions (RQ 2.a). We tested the hypothesis that the students exposed to the mastery-oriented shared control over problem selection in the Learning Phase will transfer and apply the Mastery Rule during the Future Learning Phase with the shared control. Specifically, we compared students’ problem-selection decisions between the “Mastery to Shared” condition and the “Standard to Shared” condition. In the “Mastery to Shared” condition, 15 out of 49 students (30.61 %) selected at least one mastered problem during the Future Learning Phase, whereas in the “Standard to Shared” condition, 7 out of 35 (20 %) students selected at least one mastered problem. Moreover, on average 2.7 % of the problems selected by the “Mastery to Shared” condition were mastered, while 1.6 % selected by the “Standard to Shared” condition were mastered. Nevertheless, an ANCOVA test revealed that the difference between the percentages of these two conditions was not statistically significant.
Means and SDs for mid-test and post-test equation solving items
Mastery to shared
Mastery to standard
Standard to shared
Standard to standard
4 Discussion, Conclusions and Future Work
Our classroom experiment investigated whether mastery-oriented shared control over problem selection would foster the learning of an effective problem selection strategy, students’ learning outcomes and enjoyment, as well as future problem selection and future domain-level learning. We found that shared control over problem selection, while it was supported with mastery-oriented features, led to better learning outcomes as compared to full system control in an ITS. Specifically, during the Learning Phase, those in the mastery-oriented shared control condition improved significantly more than those in the system-controlled condition on equation solving. Although the two conditions overall did not improve significantly due to the ceiling effects on the pre-test. Within the lower-performing group, there were significant learning gains from pre-test to mid-test, and the condition effect was marginally significant. These results prove that shared control accompanied by mastery-oriented features can significantly benefit students’ domain level learning, especially for students with low prior knowledge. How did the mastery-oriented shared control over problem selection lead to greater learning gains? First, the students with the mastery-oriented shared control selected almost the same problems as the system control. They rarely violated the Mastery Rule, put differently, the students selected mostly unmastered problems as the Cognitive Mastery algorithm does for the system control. Therefore, we can mostly rule out the possibility that the difference in learning gains was due to differences in the problem sequences being practiced. Second, it is likely that the mastery-oriented features (tutorial, feedback, achievements and problem selection recap screens) might have encouraged the students to adopt metacognitive strategies such as reviewing, reflecting or summarizing, as a mastery-approach orientation has been found to be positively associated with use of such strategies . Prior work has generally found that students with a mastery-approach orientation achieve better learning outcomes, compared to their counterparts who focused more on performance relative to others, i.e., with a performance orientation .
We also found that the mastery-oriented shared control resulted in significantly better declarative knowledge of the Mastery Rule, as compared to the full system control condition. It could possibly be attributed to the explicit instructions and motivational messages from the four mastery-oriented features. On the other hand, the mastery-oriented shared control did not lead to significantly higher enjoyment of using the tutor as compared to the full system-controlled tutor. It is likely that the badges, as well as the mastery bars implemented in the system-controlled condition also made it enjoyable to students. Prior work on learner control emphasizes its motivational benefits to students , but our finding suggests that enabling learner control does not necessarily enhance students’ enjoyment of the learning experience.
Although the mastery-oriented shared control enhanced students’ learning while it was in effect, no lasting effect on learning was found with only shared control over problem selection. For the Future Learning Phase, no significant condition effects were observed for learning gains on equation solving. In other words, there was apparently no carry into the next unit of a possible motivational effect on student learning. Additionally, the equations in this phase were more difficult than the Learning Phase, and the learning time was reduced to 2 class periods. The students might experience higher cognitive load when learning more difficult equations within a shorter period of time, making it difficult to initiate metacognitive processes such as reviewing or reflecting that relate to a mastery-approach orientation.
Lastly, with respect to problem selection decisions, students with shared control exhibited good application of the Mastery Rule in both phases. The mastery-oriented shared control condition selected only about 1 % of mastered problems during the Learning Phase. Similarly, the two shared control conditions without the mastery-oriented features in the Future Learning Phase selected around 2 % of mastered problems regardless of whether or not they came from the mastery-oriented shared control condition. The results regarding problem selection decisions were slightly surprising, given that in our prior classroom study, students selected 34 % mastered problems when no Open Learner Model was presented . In other prior work, we also found students admitting that they would keep selecting easy problems if given control over problem selection . There may be two reasons why students made overall good problem selection decisions in both phases: First, our informal classroom observations found that the badges and the mastery bars strongly encouraged the students to complete the levels without repeating already-mastered problems. Although these two features were designed to make the tutor more fun and reward students’ equation solving progress, not to influence problem selection, they might have motivated the students to make problem-selection decisions based on the Mastery Rule. A second reason may have been that the environments for this experiment were not entirely self-regulatory. The students were learning in their math classes and the teachers sometimes gave informal instructions such as “now you should work on the newly unlocked levels”. The students were practicing with a “goal” and supervision from their teachers, which might have influenced their problem selection decisions.
To sum up, the current experiment shows that shared control over problem selection accompanied by features that foster a mastery-approach orientation in an ITS leads to significantly better domain-level learning outcomes, as compared to full system control over problem selection, which is standard practice in ITS. This is a novel contribution to the literature on the effects of learner control on student learning, which has generally found that pure learner control leads to worse learning than system control [2, 10] and that shared control only resulted in learning outcomes that were comparable to system control . On the other hand, our experiment did not establish lasting effects of the mastery-oriented features on future learning in a new tutor unit, with improvement only on declarative knowledge of applying the rule on an immediate paper test. Future work is warranted to further investigate how to design ITSs that support learning and motivation of Self-Regulated Learning processes that can transfer to new learning topics and environments.
We thank Gail Kusbit, Jonathan Sewall, Octav Popescu and Mike Stayton for their kind help with the classroom experiment. We also thank the participating teachers and students. This work is funded by an NSF grant to the Pittsburgh Science of Learning Center (NSF Award SBE0354420).