Abstract
Prior research has shown that brief motivation interventions, such as interventions targeting students’ perceptions concerning relevance of the learning material, can have long-lasting effects on students’ motivation and performance. However, the educational contexts in which these interventions have been implemented have their own motivational affordances, such as the extent to which teachers support students’ perceptions of relevance in regular classes. According to the seed-and-soil hypothesis for the effectiveness of psychological interventions, such interventions can be seen like a “seed” that needs to be implemented in a supportive educational context (i.e., the “soil”) in order to work. Therefore, in this study we examined the interplay between a one-time relevance intervention implemented in mathematics classrooms and the mathematics teacher’s relevance support as perceived by the students before and after the intervention. Data stemmed from a cluster-randomized controlled trial with 79 ninth-grade mathematics classes (N = 1744 students) in which the intervention was implemented by the mathematics teacher or a master’s student, both trained for this purpose. Multilevel moderation models showed that both intervention conditions had larger effects when students perceived higher relevance support before or after the intervention, thus providing support for the seed-and-soil hypothesis. Furthermore, multilevel mediation models indicated a positive effect of the teacher condition on perceived relevance support at posttest compared with the control condition, which partly explained the positive effect of this intervention condition on utility value at follow-up. The results shed light on the contexts in which these interventions are most effective and the mechanisms through which they work.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
1 Introduction
In recent decades, there has been steep growth in research on motivation interventions in educational psychology. These interventions target specific psychological processes that are important sources of motivation and can have surprisingly strong and long-lasting effects on students’ motivation and performance (Lazowski & Hulleman, 2016; Yeager & Walton, 2011). One particular type of motivation intervention that has aroused great interest consists of relevance interventions, which help students see the relevance of what they are learning in school for their current and future lives (Hulleman & Harackiewicz, 2021).
However, motivation interventions are not implemented in a vacuum but in a particular instructional context that has its own motivational affordances or constraints. For instance, teachers might already support students’ perceptions of relevance to different degrees (Parrisius et al., 2020). Recently, Walton and Yeager (2020) argued that psychological interventions can work only in contexts that afford the way of thinking that is offered by the intervention—like a seed that can grow only in fertile soil. Moreover, a change in context might sometimes be necessary to yield positive effects (Walton & Yeager, 2020). When interventions are implemented in the classroom context and teachers are exposed to the intervention materials and the underlying theory, this might actually change the context through affecting teachers’ teaching beliefs and practices. Specifically, relevance interventions—though aimed at the students—might affect the teachers’ efforts to support students’ perceived relevance. Students might also interpret the context differently after participating in the intervention, regardless of actual changes in the teachers’ behavior. Such changes in students’ perceptions of their teacher might ultimately facilitate an intervention’s effects on students’ motivation.
In this study, we examined the interplay between a relevance intervention implemented in mathematics classrooms and the motivational teaching practices applied before and after the intervention in regular classes. To this end, we used data from a cluster-randomized trial with 79 mathematics classes that tested the effectiveness of a 90-min relevance intervention, which was implemented by the regular mathematics teacher or a master’s student after training (Gaspard et al., 2021). Specifically, we investigated whether perceived relevance support before and after the intervention moderated the effects of the intervention on students’ utility value, whether students’ perceptions of relevance support changed after the intervention, and whether such changes mediated effects of the intervention on students’ utility value.
1.1 Motivation interventions as a means to foster students’ motivation
Motivation interventions have received much attention in recent years. One type of intervention that has been implemented successfully in mathematics classrooms consists of relevance interventions. Relevance interventions are often grounded in Eccles et al.’s expectancy-value theory (Eccles et al., 1983), which has recently been renamed as situated expectancy-value theory (SEVT) to emphasize the situative nature of motivation (Eccles & Wigfield, 2020). SEVT posits that the most proximal predictors of students’ academic choices are their expectancies of how well they will do on specific tasks and the value they ascribe to these tasks. Eccles et al. (1983) distinguished different components that positively influence students’ subjective valuing of a given task, including its intrinsic value (enjoyment of the task), attainment value (personal importance of doing well on the task), and utility value (perceived usefulness of the task for achieving one’s goals). A large body of research has supported the basic assumptions of SEVT, showing that students’ expectancies and values in a particular domain (e.g., mathematics) are important predictors of their engagement, achievement, and academic choices in this domain (e.g., Marsh et al., 2005; Watt et al., 2012).
Intervention research drawing on SEVT has focused on utility value because it is assumed to be to be the most malleable value component (Gaspard et al., 2015a; Hulleman et al., 2010). Compared with attainment and intrinsic value, utility value is more extrinsic in nature (Eccles & Wigfield, 2020) and seems to be more easily influenced from the outside. By fostering perceived utility value, these interventions should also enhance students’ engagement and interest in the domain and, thus, ultimately lead to higher performance (Hulleman et al., 2010). Indeed, interventions targeting utility value have been shown to positively affect intrinsic and attainment value, interest, and achievement (Brisson et al., 2017; Gaspard et al., 2015a; Hulleman et al., 2010). Even though these interventions mainly target utility value, we use the term relevance intervention here to denote that they rely on mechanisms that include not only utility but also target relevance as “a personally meaningful connection to the individual” (Priniski et al., 2018, p. 12) more broadly and can lead to effects on motivational outcomes beyond utility value (see Hulleman & Harackiewicz, 2021).
The Motivation in Mathematics (MoMa) intervention is a relevance intervention that was developed for ninth-grade mathematics classes. It consists of a 90-min lesson on the relevance of mathematics as a domain, including an instructor-led psychoeducational presentation and writing tasks for individual students. We have conducted two large cluster-randomized trials to test this intervention so far. In MoMa 1, researchers who were involved in developing the intervention implemented it in the classroom. With this first trial, we found positive effects of the intervention versus a waitlist control condition on students’ values (including their utility value), expectancies, teacher-rated effort, and standardized achievement scores (Brisson et al., 2017; Gaspard et al., 2015a). To test its effectiveness under more realistic conditions for scaling up, the regular mathematics teachers or master’s students implemented the intervention in MoMa 2 after training. Both intervention conditions again showed positive effects on utility value compared with a waitlist control condition until 3 months after the intervention (d = 0.09–0.18; Gaspard et al., 2021). However, these effects were somewhat smaller compared with MoMa 1 and did not extend to all of the outcomes for which positive effects were observed in the first trial. Only small differences in the effectiveness of the intervention were observed between the teachers and master’s students, with the effects of the master’s student condition extending to some more outcomes, but no differences between the two intervention conditions were found in the effects on utility value. Although teachers and master’s students both showed a high level of implementation fidelity as indicated by the adherence rated through observers, master’s students showed an even higher level of adherence.
1.2 Motivational teaching practices in regular mathematics classes
Teachers also apply a variety of teaching practices to foster their students’ motivation in regular mathematics classes. Making what students learn in class relevant to their lives, in particular, has been discussed as a strategy for fostering students’ motivation in several theoretical frameworks in educational psychology and in mathematics education. According to self-determination theory, providing meaningful rationales and thus speaking to the relevance of a task is one possible way to support students’ need for autonomy (Su & Reeve, 2011). Indeed, research has shown that teachers’ general level of relevance support and their lesson-specific provision of relevance support are linked with different components of students’ motivation, including their utility value (Flunger et al., 2022; Parrisius et al., 2020, 2022).
In mathematics didactics, making mathematics useful has been discussed among the concepts of realistic mathematics education (van den Heuvel-Panhuizen & Drijvers, 2014) and mathematical modeling (Blum & Niss, 1991). Although increasing students’ motivation is often mentioned as one goal of these instructional approaches, they are also aimed at increasing students’ understanding and mathematical competences (Blum & Niss, 1991; Freudenthal, 1968). Indeed, a qualitative study by Pierce and Stacey (2006) revealed that teachers place high priority on students’ attitudes and therefore use real-world problems to foster students’ engagement. However, they also noted that although this strategy can make learning more enjoyable for students, it may be at the cost of more substantive learning goals. Moreover, whereas modeling problems are often used to increase student motivation, Krawitz and Schukajlow (2018) showed that students actually report lower self-efficacy and task value for modeling problems compared with other types of mathematical problems (see also Schukajlow et al., 2012). Furthermore, Rellensmann and Schukajlow (2017) found that pre-service teachers overestimate students’ interest in solving real-world problems. However, Schukajlow et al. (2012) showed that students’ self-efficacy and interest can be increased when students are taught using modeling problems for a while, particularly if student-centered teaching methods are applied.
1.3 Seed-and-soil hypothesis: interactions between motivation interventions and the motivational affordances of the broader educational context
The effects of motivation interventions on students’ academic outcomes have been found to vary (Lazowski & Hulleman, 2016; Rosenzweig & Wigfield, 2016). These interventions are assumed to trigger recursive processes in students as they engage in a particular educational context, which can then lead to long-term effects of the interventions (Harackiewicz & Priniski, 2018; Yeager & Walton, 2011). However, as detailed above, educational contexts have their own motivational affordances. Therefore, motivation interventions are viewed as context-dependent (Harackiewicz & Priniski, 2018; Yeager & Walton, 2011). Recently, Walton and Yeager (2020) proposed that such interventions work only in contexts that afford the way of thinking offered by the intervention—like a seed that grows only in fertile soil. Alternatively, it might also be plausible to expect that interventions are not needed in contexts in which students’ motivation is already supported (Rosenzweig & Wigfield, 2016) and thus would be more effective in contexts in which students’ motivation is not fostered (“parched soil”). However, the recursive changes in students’ beliefs that should be triggered through educational-psychological interventions might not be possible if the educational context does not provide opportunities for the relevant changes, which is why Walton and Yeager (2020) assume that a supportive context is needed. Although the seed-and-soil metaphor clearly has its limitations (including the implied passivity of the context), it has received much attention in intervention research and can be fruitful in understanding heterogeneity of intervention effects. Current research points towards positive interactions in line with the seed-and-soil hypothesis. For example, Yeager et al. provided support for the moderation of the effects of a growth mindset intervention through the broader school context as well as teachers’ beliefs in a large study. First, they found that the intervention worked better in schools with peer norms that supported the intervention message (i.e., support for challenge seeking; Yeager et al., 2019). Second, the intervention was also shown to work only in mathematics classrooms in which teachers endorsed a growth mindset and were therefore supportive of the intervention’s message (Yeager et al., 2022). Another study found that a purpose intervention that taught the lay theory that school is a place to develop general skills that can help oneself and others later in life increased students’ performance only when students also received a purpose-affording note from a teacher (Reeves et al., 2021).
Similarly, relevance interventions might work only in classes where the students perceive that the teacher supports their perceptions of relevance of the materials, and thus cares about their reasons for engaging with them. By contrast, if teachers undermine the intervention’s message that mathematics is useful outside of school (e.g., by explicitly stating the contrary or by not caring about students’ concerns in this regard), this might reduce the possible effect that a one-time intervention in the classroom can have on students’ beliefs in the long term. Using the data from the MoMa 1 study described above, Parrisius et al. (2021) examined whether the effects of the intervention were moderated by several motivational teaching practices (i.e., daily life examples, learning support, enthusiastic teaching). They found only a few significant moderation effects and a mixed pattern of results in terms of the direction of effects. It is not clear whether the motivational teaching practices investigated in their study matched the intervention message sufficiently to investigate the seed-and-soil hypothesis. However, before relevance interventions are implemented at scale, it is important to know in which educational contexts they work.
Furthermore, it may be necessary to complement student-targeted interventions with changes in the educational context (Walton & Yeager, 2020). Although relevance interventions are targeted at students, it is possible that teachers also change their beliefs and practices and try to support students’ perceptions of relevance more after learning about the usefulness of such an approach, thus also leading to a change in the context. Prior research has shown that teachers can successfully apply autonomy-supportive practices (e.g., providing rationales) after brief training in a specific physics teaching unit (Flunger et al., 2019). Furthermore, teachers can be trained to endorse autonomy-supportive practices more strongly and to enact these practices in their regular classes through a more extensive program that focuses on their teaching beliefs and practices more generally (Reeve & Cheon, 2021). However, prior research has not investigated whether teachers may also change their teaching practices after a one-time intervention in their classroom, such as the MoMa intervention. It is plausible that teachers might relate material back to the intervention in subsequent lessons and try to foster students’ perceptions of relevance during their teaching more generally after being exposed to exemplary methods for applying this strategy through the intervention materials. This seems particularly likely when the teachers get to deliver the intervention themselves and to participate in relevant training. Moreover, a one-time relevance intervention delivered by the teacher might already lead the students more strongly to see that the teacher cares about students’ perceptions of the relevance of the learning materials and cares to make connections with their lives. Relatedly, a values-affirmation intervention, in which students are asked to write about their broader personal values, has been found to lead students to perceive their teacher as being more interested in their out-of-school lives and as providing more care and support when the same intervention was provided by their teacher versus a researcher (Smith et al., 2021).
1.4 The present study
In this study, we wanted to examine the interplay between a one-time relevance intervention implemented in mathematics classrooms (provided by either the regular teacher or a master’s student) and students’ perceptions of the relevance support provided by their teacher in their regular classes before and after the intervention. We used both lesson-specific and general measures of relevance support to enable us to investigate the immediate changes in students’ perceptions across the lessons following the intervention as well as changes in students’ more general perceptions. More specifically, we asked the following research questions:
-
1.
Do the effects of a one-time relevance intervention vary between mathematics classes in which students perceive different levels of relevance support? To be able to understand interactions with the context better, we considered whether the teacher was perceived to support relevance already before the intervention (thus providing a ‘fertile soil’ for the intervention seed) as well as after the intervention (thus carrying forward the message planted through the intervention). We also tested whether such interactions depended on whether the intervention was delivered through the teacher or an external person.
-
2.
Are students’ perceptions of general and lesson-specific relevance support in mathematics class affected by a one-time relevance intervention, and if so, do these perceptions mediate the effects of the intervention on utility value? We expected such changes in perceived relevance support particularly if the intervention was delivered through the teacher.
2 Method
2.1 Sample and procedure
The data stemmed from the MoMa 2 study, which was conducted within ninth-grade classes in academic track schools in the German state of Baden-Württemberg from October 2017 to March 2018 (for more information about the study design, see Gaspard et al., 2021). The Ministry of Education and Cultural Affairs in Baden-Württemberg approved the study and the collection of the data. The Ethics Committee for Psychological Research at the University of Tübingen confirmed that the procedures were in line with ethical standards of research with human subjects.
A total of 70 teachers (44.2% women; age M = 38.7, SD = 9.8; years of teaching experience M = 10.4, SD = 8.5) from 28 schools participated in the study with their 78 classes (one to five classes per school). In one of the schools, one of the participating classes, which was usually comprised of the same group of students in each subject, was divided into two learning groups for mathematics that were taught by different teachers. We treated these two learning groups as separate classes in our analyses because of our focus on the teachers’ relevance support, resulting in a sample of 79 classes. Within each school, participating teachers and their classes were randomly assigned to one of two intervention conditions (intervention delivered by the mathematics teacher or a master’s student) or to a waitlist control condition, in which the intervention was implemented after the last wave of data collection.
Students’ participation was voluntary and nonincentivized, and parents and students had to provide written consent. Overall, 1744 students participated in the study (88.7% participation rate). Students’ mean age was 14.63 years (SD = 0.48) at the beginning of the study. In line with the typical composition of academic track schools, 31.7% of the students had a migration background (i.e., the student or one of their parents was not born in Germany), 72.9% had at least one parent who had obtained a general university entrance qualification, and the average socioeconomic status was relatively high (highest international socio-economic index of occupational status [HISEI] was M = 64.7 on a scale from 16 to 90).
The study consisted of several waves of data collection. Students were administered questionnaires by trained research assistants before the intervention (pretest = T1), an average of 4 weeks after the intervention (posttest = T2), and an average of 3 months after the intervention (follow-up = T3). Additionally, students’ lesson-specific perceptions were assessed during the five mathematics lessons following the intervention. These brief repeated questionnaires were administered by the teachers at the end of each lesson. In one of the classes, the allocation of individual questionnaires was not implemented correctly because the teacher failed to hand the questionnaires out to the students as indicated by a post-it note on the questionnaire, and this incorrect allocation could not be resolved later. Thus, the data from this class are missing for the lesson-specific questionnaires.
2.2 MoMa relevance intervention in the classroom
The intervention was a 90-min lesson on the relevance of mathematics, which consisted of an instructor-led psychoeducational presentation for the whole class (~ 45 min) and a relevance-inducing task that the students worked on individually (~ 40 min; for more information, see Gaspard et al., 2021). The presentation included various examples of the utility of mathematics for future education, career opportunities in different fields, and leisure time activities. In the individual task, students were asked to read and evaluate interview quotations of young adults describing situations in which mathematics was useful to them. Although some of the examples referred to specific content, the intervention as a whole was targeted to the relevance of mathematics as a broader domain.
The master’s students and teachers who implemented the intervention in the classroom received all intervention materials from the research team and these materials did not differ between conditions (for more information about the training in both conditions, see Gaspard et al., 2021).
A total of six master’s students (five women and one man; age M = 24.7, SD = 1.5) delivered the intervention in the master’s student condition (four to five classes per student). They were trained for this purpose as part of a two-semester class on motivation interventions in the master’s program Education Sciences and Psychology at the University of Tübingen.
The teachers (n = 24; 45.8% women; age M = 40.2, SD = 9.8; years of teaching experience M = 11.8, SD = 8.9) assigned to the teacher condition were asked to participate in a 3-h workshop in small groups to prepare for the intervention implementation. One teacher in this condition declined to participate in the workshop and thus did not deliver the intervention. We followed the intention-to-treat approach in our analyses and included this class in the teacher condition so that the random assignment was kept intact. In the workshops, the teachers were provided with brief information about the theoretical background of the intervention, the importance of conducting randomized experiments, and the results of the MoMa 1 trial. The teachers were then walked through the intervention, followed by a discussion about potential challenges when delivering the intervention in the classroom.
Two trained observers attended each intervention and rated the degree of implementation fidelity. Overall, the intervention was implemented as planned in both intervention conditions, with the observers noting only very few severe deviations from the script. All predefined intervention phases were implemented in all classes. Moreover, adherence was rated to be high in both conditions (M = 7.29 and 9.05 in the teacher and master’s student conditions, respectively, on a scale from 1 to 10).
2.3 Instruments
All items were rated on a 4-point Likert scale ranging from 1 (completely disagree) to 4 (completely agree). The mean scores for the respective scales were used in the analyses. An overview of the instruments is shown in Fig. 1.
2.3.1 Utility value
Mathematics utility value was the primary outcome of the intervention and was assessed at T1, T2, and T3. For the purpose of this study, we slightly adapted a scale developed by Gaspard et al. (2015b) and further improved by Gaspard et al. (2017). A total of 12 items tapping general utility (e.g., “Math is very useful to me”), utility for job (e.g., “A good knowledge of math will help me in my future job”), utility for daily life (e.g., “Knowing about the subject of math brings me many advantages in my daily life”), and utility for school (e.g., “Being good at math will help me in the remaining school years”) were used. In this study, we were interested in the overall level of utility value and thus combined all items into one mean score (see Gaspard et al., 2021, for effects of the intervention conditions on the different subscales). The scale had high internal consistency at all time points (α = .88–.89).
2.3.2 Relevance support
Students were asked to report their general perceptions of relevance support from their mathematics teacher with five items at T1 and T2 (e.g., “I feel that in general in our math lessons, we are shown how the learning content is personally relevant to us”). These items were partly taken from a scale measuring the provision of meaningful rationales (Flunger et al., 2019) and partly newly developed to refer to the usefulness of mathematics in everyday and future life specifically. The scale’s internal consistency was high at both time points (α = .82 and .87). Furthermore, students were asked to rate their perceived lesson-specific relevance support during the five lessons following the intervention with two items (e.g., “Our teacher explained to us why the topic of today’s lesson can be relevant to us”). Considering the low number of items, the internal consistency of this scale was acceptable across time points (α = .68–.74).
2.3.3 Covariates
Similarly to the analyses performed by Gaspard et al. (2021), we used several covariates assessed at pretest in our analyses. These included students’ gender and previous mathematics grades based on information provided by the schools. Furthermore, we considered students’ self-reported intrinsic value, cost, self-concept, self-efficacy, and effort in mathematics. A brief, standardized mathematics test measured students’ fluency in solving typical mathematical operations (taken from Schmidt et al., 2013). Finally, teachers rated each individual students’ effort in mathematics on two items. More information about these scales can be accessed at https://osf.io/tr8cw/.
2.4 Statistical analyses
We conducted multilevel moderation and mediation analyses in Mplus 8.4 (Muthén & Muthén, 1998–2017) to address our research questions. All analyses considered the student and class levels. Moreover, the analyses including relevance support as assessed during the five lessons following the intervention considered this variation across time points within students at the lowest level, resulting in a three-level model. However, we focused on the student and class levels, which were set up in parallel with the two-level analyses. All continuous variables were standardized before we ran the analyses so that the regression coefficients could be understood as standardized effects with respect to the total variance in the outcome variables (Marsh et al., 2009). Because of the high sampling ratio, manifest aggregation was used for all predictor variables (Marsh et al., 2009). We used grand-mean centering to be able to estimate the cross-level mediation paths (Pituch & Stapleton, 2012). To deal with missing data, we used the full information maximum likelihood method, which considers all available information.
2.4.1 Multilevel moderation analyses
Multilevel regression analyses were conducted to test whether the effects of the intervention on utility value at T2 and T3 were moderated by the teachers’ relevance support before and after the intervention (RQ1). In these analyses, we controlled for utility value at T1 and a set of covariates at the student and class levels to yield more precise estimates of the intervention effects (Raudenbush, 1997). In line with the guidelines proposed by the What Works Clearinghouse (2020), we included those variables for which we found small differences between the experimental conditions before the intervention (0.05 < d < 0.20) in our analyses in addition to the pretest score (i.e., gender, grade, intrinsic value, cost, self-concept, self-efficacy, effort, achievement score, and teacher-rated effort; see Gaspard et al., 2021, for the pretest differences). At the class level, the two intervention conditions were included as dummy variables. Additionally, as predictors, we included students’ relevance support at the student and class levels and two product terms indicating the interactions between relevance support and the two intervention conditions at the class level.
2.4.2 Multilevel mediation analyses
We specified cross-level mediation models (Pituch & Stapleton, 2012) to investigate the links between the intervention implemented at the class level and utility value as a student level outcome as mediated by relevance support measured at the student level (RQ2). In contrast to class-level-only mediation approaches, cross-level mediation models consider both the individual and shared perceptions of students when testing for mediational associations. Such models thereby provide information about (a) whether the class-level predictor has an indirect effect on the student-level outcome through a student-level mediator (cross-level indirect effect) and (b) whether there is an additional indirect effect of the class-level predictor on the outcome over and above the indirect effect via the student-level mediator (unique class-level indirect effect). We thus tested whether the two intervention conditions affected students’ relevance support at the class level and whether there were indirect effects of the two intervention conditions on utility value through relevance support at both the student level (cross-level indirect effect) and the class level (unique class-level indirect effect, see Fig. 2). The total indirect effect was then computed as the sum of the cross-level and the unique class-level indirect effects. We included relevance support and utility value at T1 as predictors on the student and class levels to investigate changes in these variables, and additionally controlled for the set of covariates described above. In line with the moderation analyses, we also included potential interactions between relevance support at pretest and the two intervention conditions.
3 Results
3.1 Descriptive statistics
Descriptive statistics for the major study variables are presented in Table 1. Correlations between all study variables can be found at https://osf.io/tr8cw/.
3.2 Moderation analyses for testing relevance support as a moderator of the intervention effects on utility value
We tested a total of six moderation models involving utility value as the outcome assessed at T2 or T3 and relevance support as the moderator assessed at T1, at T2, or during the five lessons after the intervention. The main results are reported in Table 2; full models including all covariates and the variance components at both levels can be found online (https://osf.io/tr8cw/). Across all models, relevance support showed positive effects on utility value at the student level. Because we used grand-mean centering, the effects of relevance support at the class level represent additional effects of the class composition beyond student-level effects. Controlling for students’ individual perceptions of relevance support, there was a tendency for a negative effect of being in a class with high shared perceptions of relevance support.
Concerning relevance support at T1 as a moderator, we found a significant, positive interaction between the teacher condition and relevance support for utility value at T2 as an outcome. That is, larger effects of the intervention were observed for classes in which the students had already perceived high relevance support before the intervention.
For relevance support at T2 as the moderator, we found a marginally significant positive interaction with the master’s student condition for utility value at T2 and a significant positive interaction with the master’s student condition for utility value at T3. Both interactions can be interpreted to mean that the intervention was more effective in classrooms in which students also generally perceived higher relevance support from their teacher after the intervention. Finally, for relevance support during the five mathematics lessons following the intervention, we again observed a significant positive interaction with the master’s student condition but only at T3.
3.3 Mediation models for testing indirect effects of the intervention on utility value through relevance support
We tested two mediation models, namely, (a) relevance support at T2 as a mediator of intervention effects on utility value at T3 (see Table 3), and (b) relevance support during the five lessons after the intervention as a mediator of intervention effects on utility value at T2 (see Table 4, for the full models with all covariates and variance components see https://osf.io/tr8cw/). With respect to relevance support at T2, we found that—compared with the control condition—the teacher condition had a positive effect on students’ relevance support at T2, which, in turn, positively predicted utility value at T3 at the student level. No further contextual effect of relevance support (i.e., an effect of students’ shared perceptions of relevance support when controlling for their individual perceptions) on utility value was found. These paths resulted in a significant cross-level indirect effect and a marginally significant total indirect effect of the teacher condition. For the master’s student condition, we observed only a marginally significant direct effect of the intervention (vs. the control condition) on students’ utility value but no effects involving relevance support. These mediated paths were not moderated by relevance support at T1.
For relevance support during the five lessons after the intervention, there were no significant effects of the two intervention conditions on relevance support. However, relevance support during these lessons predicted utility value at T2 at the student level. The model also showed significant effects of both intervention conditions on utility value at T2, which could not be explained by relevance support during the five lessons after the intervention. Thus, no significant indirect effects could be observed.
4 Discussion
In the present study we examined whether the effects of a one-time relevance intervention were moderated by perceived relevance support before and after the intervention and whether the intervention had indirect effects on students’ utility value through their perceived relevance support. Data stemmed from a large cluster-randomized effectiveness trial in which the intervention was implemented by either the regular mathematics teacher or a master’s student. Interactions between the two intervention conditions and relevance support before and after the intervention provided support for the seed-and-soil hypothesis (Walton & Yeager, 2020): Larger intervention effects tended to be found in classes where students perceived stronger relevance support from their teachers during regular classes and, thus, where the teachers afforded the intervention’s message. We also found some support for indirect effects of the relevance intervention on utility value through students’ perceptions of relevance support after the intervention, but only if the intervention was provided by their own teacher, which sheds light on the mechanisms at work in this condition.
4.1 Relevance support as a contextual moderator of intervention effects
We found some evidence for our assumption that intervention effects would be moderated by students’ perceived relevance support in the classroom in line with the seed-and-soil hypothesis (Walton & Yeager, 2020). These effects varied between intervention conditions and the time at which the moderator was measured. In the master’s student condition, interactions were mostly found for relevance support after the intervention, with similar patterns for general perceptions at T2 and lesson-specific perceptions during the lessons following the intervention. When the intervention was delivered by an external person, the intervention was thus more effective if its message was reinforced (and not undermined) by the teacher afterwards. These findings are similar to results of prior research that reported that computerized motivation interventions are effective only when their message is afforded later by the teacher (Reeves et al., 2021). For the teacher condition, we found a significant positive interaction only for perceived relevance support before the intervention. Here, it thus seemed to matter more whether the teacher had already provided a fertile soil before the intervention, which might have resulted in a more authentic delivery of the intervention message through the teacher. In contrast to our study, Parrisius et al. (2021) found only a few moderation effects in the MoMa 1 study, where the intervention was delivered by researchers, which is closer to the master’s student condition in our study. However, the motivational practices considered as moderators in their study might not have been close enough to the intervention message, whereas the moderator considered here (i.e., perceived relevance support) is arguably a direct assessment of the psychological affordance of the intervention message through the instructional context. Moreover, they only considered students’ perceptions of motivational practices prior to the intervention in their study.
In general, all interaction terms found in the current study were positive, in line with the seed-and-soil hypothesis. Under some circumstances, negative interactions indicating that interventions are most effective in contexts in which students’ motivation is not supported already (‘parched soil’) might also be found. However, it seems that a negative development such as a decline in utility value in mathematics during secondary school (Gaspard et al., 2017) can best be prevented through the combination of a targeted intervention and a supportive instructional context. These varying effects across contexts are also important to consider when implementing student-targeted interventions in educational practice. The intervention approach might have to be changed in instructional contexts in which teachers do not successfully support students’ perceptions of relevance during regular lessons. Some educational contexts might be too ‘parched’ to afford positive motivational development and might require other types of interventions focused on changing the context (e.g., targeting teachers’ beliefs and practices).
4.2 Relevance support as an intervention mechanism in the teacher condition
Students in classes in which the intervention was implemented by their teacher reported higher relevance support at posttest, which could partly explain the positive effects of the teacher condition on utility value at follow-up. This result is interesting insofar as similar effects of the two intervention conditions were observed although the master’s students showed a higher adherence to the script when delivering the intervention (Gaspard et al., 2021). It thus seems that teachers achieved the same end through different means. Our findings are similar to those of Smith et al. (2021), who found that students perceived that their teacher provided more care and support when they received a teacher-provided (vs. a researcher-provided) values-affirmation intervention. Two explanations for higher perceived relevance support at posttest are possible. Teachers could have tried to relate the material back to the intervention message in the lessons after the intervention and would thereby have supported students’ perceived relevance to a greater extent. Alternatively, the one-time delivery of the intervention, which was quite different from a regular mathematics lesson, might have already been enough to change students’ perceptions of relevance support from their teacher. The items assessing perceived relevance support at posttest were formulated in a general way, which makes the first explanation more plausible. However, although the regression coefficient for the intervention effect on relevance support during the five lessons that followed the intervention was also positive, it was not statistically significant, so that there was no difference in lesson-specific relevance support between the teacher and the control conditions. This finding instead supports the second explanation. More research is thus needed to examine whether a brief training session and the delivery of a scripted intervention are enough to trigger actual changes in teachers’ teaching beliefs and practices during regular classes following this intervention. Still, our findings speak to the potential power of having teachers deliver motivation interventions in educational practice as this may positively affect students’ perceptions of their teacher. To maximize effects of motivation interventions, it might be necessary not only to target students’ motivation but also to involve teachers in the intervention approach and change their teaching beliefs and practices in ways that support students’ motivation (Reeve & Cheon, 2021). Our study, however, applied a more minimal approach in that teachers participated only in a short workshop focused on delivering a one-time relevance intervention rather than changing their teaching beliefs and practices more generally. When it comes to developing interventions targeted at teachers, it might also be important to consider individual teachers’ beliefs and practices—some might need different intervention techniques than others.
4.3 Limitations
Although we used data from a large cluster-randomized trial in which the intervention was delivered under conditions that were close to educational practice, our study also has several limitations that need to be considered when interpreting its findings. First, our sample was limited to ninth-grade students in the academic track in one region in Germany. Future research therefore needs to examine whether our findings generalize across grade levels, school types, and regions. Second, when interpreting the effects of the intervention, we referred to effects in comparison with the control condition. However, these results are not necessarily aligned with the mean-level trajectory over time within the conditions. In terms of this trajectory, we observed a decline in utility value in the control condition, as has been found in previous research (e.g., Gaspard et al., 2015a). Instead of lifting utility value overall, the intervention thus functioned as a buffer against this negative development. The effects that we found were furthermore small relying on conventional standards, but can be judged as medium to large when using the benchmarks proposed for educational interventions and considering the low cost of the intervention (Kraft, 2020). Third, whereas the study was adequately powered to detect small (and realistic) main effects (δ = 0.20; see Gaspard et al., 2021), the power to detect effects of cluster-level moderators, such as the ones examined in our study, was likely lower (Parrisius et al., 2021; Spybrook et al., 2016). It is therefore unclear whether the effects we found capture all true moderation effects or whether they overestimate the true effects. A replication with more classes is necessary to be able to reliably detect small moderation effects. Fourth, we focused on utility value as the primary outcome of the intervention, but future research should also investigate whether similar interactions can be found for more distal outcomes such as academic choices and achievement. Finally, we considered relevance support only as perceived by the students. Although the students’ perspective is arguably most important for the psychological affordance of the instructional context, this prevented us from coming to a clear conclusion about whether teachers changed their teaching beliefs and practices after the intervention or whether students just perceived them differently. It would be necessary to incorporate teacher and potentially observer ratings to fully answer this question.
5 Conclusions
The present study sheds light on the interactions between a one-time relevance intervention implemented in the classroom and teachers’ continuing instructional practices to support perceived relevance. Our study has two main conclusions. First, if the interventions are delivered by external individuals, they are more beneficial for students’ motivation if the teachers also afford the intervention message afterwards. Second, when teachers deliver the interventions in the classroom, this can promote students’ perceptions of their teachers’ relevance support, which further helps explain effects on students’ valuing of mathematics later in the school year. Our findings thus contribute to knowledge about the conditions in which such interventions work and how such interventions can be implemented more effectively in practice. Future research will need to continue examining motivation interventions in diverse contexts using adequate research designs to be able to test the robustness and generalizability of these findings.
Data availability statement
The data set analyzed during the current study are available from the corresponding author on request.
References
Blum, W., & Niss, M. (1991). Applied mathematical problem solving, modelling, applications, and links to other subjects—State, trends, and issues in mathematics instruction. Educational Studies in Mathematics, 22(1), 37–68. https://doi.org/10.1007/BF00302716
Brisson, B. M., Dicke, A.-L., Gaspard, H., Häfner, I., Flunger, B., Nagengast, B., & Trautwein, U. (2017). Short intervention, sustained effects: Promoting students’ math competence beliefs, effort, and achievement. American Educational Research Journal, 54(6), 1048–1078. https://doi.org/10.3102/0002831217716084
Eccles, J. S., Adler, T. F., Futterman, R., Goff, S. B., Kaczala, C. M., Meece, J. L., & Midgley, C. (1983). Expectancies, values, and academic behaviors. In J. T. Spence (Ed.), Achievement and achievement motives (pp. 74–146). W. H. Freeman.
Eccles, J. S., & Wigfield, A. (2020). From expectancy-value theory to situated expectancy-value theory: A developmental, social cognitive, and sociocultural perspective on motivation. Contemporary Educational Psychology, 61, 101859. https://doi.org/10.1016/j.cedpsych.2020.101859
Flunger, B., Hollmann, L., Hornstra, L., & Murayama, K. (2022). It’s more about a lesson than a domain: Lesson-specific autonomy support, motivation, and engagement in math and a second language. Learning and Instruction, 77, 101500. https://doi.org/10.1016/j.learninstruc.2021.101500
Flunger, B., Mayer, A., & Umbach, N. (2019). Beneficial for some or for everyone? Exploring the effects of an autonomy-supportive intervention in the real-life classroom. Journal of Educational Psychology, 111(2), 210–234. https://doi.org/10.1037/edu0000284
Freudenthal, H. (1968). Why to teach mathematics so as to be useful. Educational Studies in Mathematics, 1(1/2), 3–8.
Gaspard, H., Dicke, A.-L., Flunger, B., Brisson, B. M., Häfner, I., Nagengast, B., & Trautwein, U. (2015a). Fostering adolescents’ value beliefs for mathematics with a relevance intervention in the classroom. Developmental Psychology, 51(9), 1226–1240. https://doi.org/10.1037/dev0000028
Gaspard, H., Dicke, A.-L., Flunger, B., Schreier, B., Häfner, I., Trautwein, U., & Nagengast, B. (2015b). More value through greater differentiation: Gender differences in value beliefs about math. Journal of Educational Psychology, 107(3), 663–677. https://doi.org/10.1037/edu0000003
Gaspard, H., Häfner, I., Parrisius, C., Trautwein, U., & Nagengast, B. (2017). Assessing task values in five subjects during secondary school: Measurement structure and mean level differences across grade level, gender, and academic subject. Contemporary Educational Psychology, 48, 67–84. https://doi.org/10.1016/j.cedpsych.2016.09.003
Gaspard, H., Parrisius, C., Piesch, H., Kleinhansl, M., Wille, E., Nagengast, B., Trautwein, U., & Hulleman, C. S. (2021). The potential of relevance interventions for scaling up: A cluster-randomized trial testing the effectiveness of a relevance intervention in math classrooms. Journal of Educational Psychology, 113(8), 1507–1528. https://doi.org/10.1037/edu0000663
Harackiewicz, J. M., & Priniski, S. J. (2018). Improving student outcomes in higher education: The science of targeted intervention. Annual Review of Psychology, 69, 409–435. https://doi.org/10.1146/annurev-psych-122216-011725
Hulleman, C. S., Godes, O., Hendricks, B. L., & Harackiewicz, J. M. (2010). Enhancing interest and performance with a utility value intervention. Journal of Educational Psychology, 102(4), 880–895. https://doi.org/10.1037/a0019506
Hulleman, C. S., & Harackiewicz, J. M. (2021). The utility-value intervention. In G. M. Walton & A. J. Crum (Eds.), Handbook of wise interventions: How social psychology can help people change (pp. 100–125). Guilford Press.
Kraft, M. A. (2020). Interpreting effect sizes of education interventions. Educational Researcher, 49(4), 241–253. https://doi.org/10.3102/0013189X20912798
Krawitz, J., & Schukajlow, S. (2018). Do students value modelling problems, and are they confident they can solve such problems? Value and self-efficacy for modelling, word, and intra-mathematical problems. ZDM, 50(1–2), 143–157. https://doi.org/10.1007/s11858-017-0893-1
Lazowski, R. A., & Hulleman, C. S. (2016). Motivation interventions in education: A meta-analytic review. Review of Educational Research, 86(2), 602–640. https://doi.org/10.3102/0034654315617832
Marsh, H. W., Lüdtke, O., Robitzsch, A., Trautwein, U., Asparouhov, T., Muthén, B., & Nagengast, B. (2009). Doubly-latent models of school contextual effects: Integrating multilevel and structural equation approaches to control measurement and sampling error. Multivariate Behavioral Research, 44(6), 764–802. https://doi.org/10.1080/00273170903333665
Marsh, H. W., Trautwein, U., Lüdtke, O., Köller, O., & Baumert, J. (2005). Academic self-concept, interest, grades, and standardized test scores: Reciprocal effects models of causal ordering. Child Development, 76(2), 397–416. https://doi.org/10.1111/j.1467-8624.2005.00853.x
Muthén, L. K., & Muthén, B. O. (1998). Mplus user’s guide (8th ed.). Muthén & Muthén.
Parrisius, C., Gaspard, H., Flunger, B., Trautwein, U., & Nagengast, B. (2021). Gleiche Wirkung in jedem Klassenzimmer? Moderationseffekte durch motivationale Unterrichtspraktiken am Beispiel einer Nützlichkeitsintervention im Mathematikunterricht und damit einhergehende Herausforderungen [Same effect in every classroom? Treatment by moderator effects of a relevance intervention as a function of motivational teaching practices, and methodological challenges]. In R. Lazarides & D. Raufelder (Eds.), Motivation in unterrichtlichen Lehr-Lernkontexten [motivation in classroom teaching and learning contexts] (pp. 355–388). Springer.
Parrisius, C., Gaspard, H., Trautwein, U., & Nagengast, B. (2020). The transmission of values from math teachers to their ninth-grade students: Different mechanisms for different value dimensions? Contemporary Educational Psychology, 62, 101891. https://doi.org/10.1016/j.cedpsych.2020.101891
Parrisius, C., Gaspard, H., Zitzmann, S., Trautwein, U., & Nagengast, B. (2022). The “situative nature” of competence and value beliefs and the predictive power of autonomy support: A multilevel investigation of repeated observations. Journal of Educational Psychology, 114(4), 791–814. https://doi.org/10.1037/edu0000680
Pierce, R., & Stacey, K. (2006). Enhancing the image of mathematics by association with simple pleasures from real world contexts. ZDM, 38(3), 214–225. https://doi.org/10.1007/BF02652806
Pituch, K. A., & Stapleton, L. M. (2012). Distinguishing between cross- and cluster-level mediation processes in the cluster randomized trial. Sociological Methods and Research, 41(4), 630–670. https://doi.org/10.1177/0049124112460380
Priniski, S. J., Hecht, C. A., & Harackiewicz, J. M. (2018). Making learning personally meaningful: A new framework for relevance research. Journal of Experimental Education, 86(1), 11–29. https://doi.org/10.1080/00220973.2017.1380589
Raudenbush, S. W. (1997). Statistical analysis and optimal design for cluster randomized trials. Psychological Methods, 2(2), 173–185. https://doi.org/10.1037/1082-989X.2.2.173
Reeve, J., & Cheon, S. H. (2021). Autonomy-supportive teaching: Its malleability, benefits, and potential to improve educational practice. Educational Psychologist, 56(1), 54–77. https://doi.org/10.1080/00461520.2020.1862657
Reeves, S. L., Henderson, M. D., Cohen, G. L., Steingut, R. R., Hirschi, Q., & Yeager, D. S. (2021). Psychological affordances help explain where a self-transcendent purpose intervention improves performance. Journal of Personality and Social Psychology, 120(1), 1–15. https://doi.org/10.1037/pspa0000246
Rellensmann, J., & Schukajlow, S. (2017). Does students’ interest in a mathematical problem depend on the problem’s connection to reality? An analysis of students’ interest and pre-service teachers’ judgments of students’ interest in problems with and without a connection to reality. ZDM, 49(3), 367–378. https://doi.org/10.1007/s11858-016-0819-3
Rosenzweig, E. Q., & Wigfield, A. (2016). STEM motivation interventions for adolescents: A promising start, but further to go. Educational Psychologist, 51(2), 146–163. https://doi.org/10.1207/s15326985ep2003
Schmidt, S., Ennemoser, M., & Krajewski, K. (2013). Deutscher Mathematiktest für 9. Klassen [German mathematics test for Grade 9]. Hogrefe.
Schukajlow, S., Leiss, D., Pekrun, R., Blum, W., Müller, M., & Messner, R. (2012). Teaching methods for modelling problems and students’ task-specific enjoyment, value, interest and self-efficacy expectations. Educational Studies in Mathematics, 79(2), 215–237. https://doi.org/10.1007/s10649-011-9341-2
Smith, E. N., Rozek, C. S., Manke, K. J., Dweck, C. S., & Walton, G. M. (2021). Teacher- versus researcher-provided affirmation effects on students’ task engagement and positive perceptions of teachers. Journal of Social Issues, 77(3), 751–768. https://doi.org/10.1111/josi.12417
Spybrook, J., Kelcey, B., & Dong, N. (2016). Power for detecting treatment by moderator effects in two- and three-level cluster randomized trials. Journal of Educational and Behavioral Statistics, 41(6), 605–627. https://doi.org/10.3102/1076998616655442
Su, Y. L., & Reeve, J. (2011). A meta-analysis of the effectiveness of intervention programs designed to support autonomy. Educational Psychology Review, 23(1), 159–188. https://doi.org/10.1007/s10648-010-9142-7
van den Heuvel-Panhuizen, M., & Drijvers, P. (2014). Realistic mathematics education. In S. Lerman (Ed.), Encyclopedia of mathematics education (pp. 521–525). Springer. https://doi.org/10.1007/978-94-007-4978-8_170
Walton, G. M., & Yeager, D. S. (2020). Seed and soil: Psychological affordances in contexts help to explain where wise interventions succeed or fail. Current Directions in Psychological Science, 29(3), 219–226. https://doi.org/10.1177/0963721420904453
Watt, H. M. G., Shapka, J. D., Morris, Z. A., Durik, A. M., Keating, D. P., & Eccles, J. S. (2012). Gendered motivational processes affecting high school mathematics participation, educational aspirations, and career plans: A comparison of samples from Australia, Canada, and the United States. Developmental Psychology, 48(6), 1594–1611. https://doi.org/10.1037/a0027838
What Works Clearinghouse. (2020). What Works Clearinghouse standards handbook. US Department of Education, Institute of Education Sciences, National Center for Education and Regional Assistance. https://doi.org/10.1037/e578392011-004
Yeager, D. S., Carroll, J. M., Buontempo, J., Cimpian, A., Woody, S., Crosnoe, R., Muller, C., Murray, J., Mhatre, P., Kersting, N., Hulleman, C., Kudym, M., Murphy, M., Duckworth, A. L., Walton, G. M., & Dweck, C. S. (2022). Teacher mindsets help explain where a growth-mindset intervention does and doesn’t work. Psychological Science, 33(1), 18–32. https://doi.org/10.1177/09567976211028984
Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., Paunesku, D., Romero, C., Flint, K., Roberts, A., Trott, J., Iachan, R., Buontempo, J., Man Yang, S., Carvalho, C. M., & Dweck, C. S. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573(7774), 364–369. https://doi.org/10.1038/s41586-019-1466-y
Yeager, D. S., & Walton, G. M. (2011). Social-psychological interventions in education: They’re not magic. Review of Educational Research, 81(2), 267–301. https://doi.org/10.3102/0034654311405999
Acknowledgements
Cora Parrisius is now affiliated with the University of Education Karlsruhe. This research was supported by the Eliteprogramme for Postdocs of the Baden-Württemberg Stiftung and the Institutional Strategy of the University of Tübingen (Deutsche Forschungsgemeinschaft, ZUK 63). Cora Parrisius was a doctoral student at the LEAD Graduate School and Research Network [GSC1028], which was funded within the framework of the Excellence Initiative of the German federal and state governments.
Funding
Open Access funding enabled and organized by Projekt DEAL.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Gaspard, H., Parrisius, C., Nagengast, B. et al. Understanding the interplay between targeted motivation interventions and motivational teaching practices in mathematics classrooms. ZDM Mathematics Education 55, 345–358 (2023). https://doi.org/10.1007/s11858-022-01446-3
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11858-022-01446-3