Phase 1: item selection
The four factors targeted by the Motivation for Mathematics Abbreviated Instrument (MMAI) are intrinsic motivation, mastery orientation, performance orientation, and expectancy. The development of MMAI began with the creation of a large pool of items from several previously published instruments measuring disparate theories of motivation. We chose these specific instruments (listed in the next few paragraphs) because of their widespread popularity in educational research. By including items from instruments with a strong record of utility in research, we began with some inherent evidence for validity of content.
To represent self-determination theory, we selected all items representing interest and enjoyment from the Intrinsic Motivation Inventory (McAuley et al. 1987). This article has been cited at least 1988 times. All items representing intrinsic and extrinsic motivation from the Sport Motivation Survey (Pelletier et al. 1995) were included, and this article has been cited at least 1948 times. Items representing intrinsic motivation from Pintrich et al.’s (1992) Motivated Strategies for Learning Questionnaire were also included. This resulted in the inclusion of 34 items representing self-determination theory. To represent achievement goal theory, we selected all items representing mastery orientation and performance orientation from the Patterns of Adaptive Learning Scale (Midgley et al. 2000). This article has been cited at least 1527 times. This resulted in the inclusion of 11 items representing mastery orientation and 10 items representing performance orientation. To represent expectancy value, we selected all items intended to measure expectancy from the Motivated Strategies for Learning Questionnaire (Pintrich et al. 1992). This article has been cited at least 1339 times. We also included the items from this survey representing task value as well as items from a Perceived Instrumentality Survey by Miller et al. (1999). This resulted in the inclusion of 12 items representing expectancy and 11 items representing task value.
Self-efficacy is the belief that you can be successful in a limited domain as described by Bandura (1997). It is a strong predictor of success and is closely related to motivation. To represent self-efficacy, we included the 24 items from Usher and Pajares’ (2009) Sources of Self-Efficacy survey.
As the intent was to measure motivation for mathematics, the roots of the items were left unchanged with only slight modifications to focus the items on mathematics. For example, “Learning this material is enjoyable” from Miller et al. (1999, p. 255) was changed to “Learning math is enjoyable”. Other than focussing the item on our intended domain, these changes had little impact on the content of the item.
Phase 2: expert review
After assembling the initial pool of 107 items, we sent emails to all 511 members of the AERA SIG for Motivation in Education asking them to evaluate the items in these surveys. These emails informed members that the purpose of the study was to develop and initially validate an abbreviated instrument intended to measure constructs from several of the predominant theories of motivation. The members were informed that the intended population was post-secondary students in the United States, and that the items utilized a five-point scale. We attached an informed consent form to these emails explaining the voluntary and anonymous nature of participation in this study. These emails also contained a request for any questions, comments, or concerns about the study.
We asked the members to select the five items that best represented each of the seven intended constructs. The items also needed to discriminate between constructs. We also asked for commentary. We received 123 responses (24% response rate) to the survey. We then ranked the items based on the members’ endorsements and analysed the comments of the item reviewers. Based on these data, 41 items representing intrinsic motivation, extrinsic motivation, mastery orientation, performance orientation, self-efficacy, expectancy, and task value became the preliminary instrument, and all had at least 40% endorsement.
In comments concerning the intrinsic motivation items, a few SIG members highlighted difficulties in separating intrinsic motivation from mastery; however, many of the comments suggested that the variable separating these constructs was enjoyment of a task. As one member wrote, “I think intrinsic motivation is all about the joy someone gets from a task, just because it is that task”, and another wrote, “intrinsic motivation speaks to something being fun and enjoyable just because”. Another SIG member wrote, “I had difficulty with doing things for some pleasure as [being] directly intrinsic, but [the intrinsic motivation items] can be reduced to five with the following words: satisfying, enjoyment, pleasure, excitement and fun”. One of the intrinsic motivation items in the MMAI references fun, one references enjoyment, one references excitement, and one references interest.
Many of the comments concerning the mastery orientation items discussed improving and learning. One member wrote, “Mastery orientation is defined as improving”, and another wrote, “Mastery orientations are expressed when a person tends to set objectives of learning as much as possible”. Another member had a slightly different view of mastery. “The focus, in my opinion, should be about the task-based standards of mastery and understanding, and less so on the self-improvement standard”. Two of the mastery orientation items in the MMAI reference learning, one references understanding, and one makes specific reference to mastery.
Comments about performance orientation items tended to focus on being better than others. As one member wrote, “Here it is important to decide whether you want to capture via the performance approach (PA) goal items; (a) the aim of being better than others; (b) the aim of demonstrating one's superiority; (c) both… I favor (a) over (b) in defining PA goals”. All the performance orientation items in the MMAI focus on performing better than others.
Regarding most of the expectancy items members stated that the items were confounded with self-efficacy. As one wrote, “I have checked the ones most like expectancy, which is actually the same thing as self-efficacy”. Thus, the commentary about the content of the included expectancy items revealed the strong relationship between expectancy and self-efficacy. This relationship is aligned with the expectancy-value framework espoused by Eccles and Wigfield (1995).
Phase 3: cognitive interviews
We used cognitive interviews (Tourangeau 1984; Willis 2004) with four students and two instructors to obtain feedback about the clarity and alignment of the 41 items with the four theoretical constructs underlying the MMAI. Unlike the theoretical nature of the feedback from experts, the cognitive interview data obtained from students and instructors were from the target population being measured.
Students were recruited from one math class at a public university in the southern United States. We began by asking students to fill out cards with their names, gender, and a response on a scale of one to ten to a question about whether they liked mathematics. From these cards, we used stratified sampling to select participants so that we had equal representation based on gender and liking mathematics. This resulted in one male who liked mathematics and one who did not—let us call them Bill and Ted, and one female who liked mathematics, and one who did not—let us call them Thelma and Louise. Two instructors were also interviewed.
Students were asked the following questions for each item:
-
1.
What was the meaning of (insert item)?
-
2.
What information did you need to answer (insert item)?
-
3.
What judgements did you make when answering (insert item)?
-
4.
What was your answer to (insert item)?
Instructors were asked similar questions; however, questions 2 and 3 were modified to ask what information or judgements a student might need to answer. These interviews provided over four hours of audio recordings.
We then transcribed these interviews, and two researchers separated these transcripts into concept phrases. For this process each comment was discussed, and when phrases within a comment could stand alone as distinct ideas, then the comment was separated into individual concept phrases. Then, we entered these concept phrases row by row into an intra-respondent matrix (Onwuegbuzie 2003). This resulted in 2764 rows of data. We then coded these concept phrases using a constant comparison methodology (Straus and Corbin 1990). In the coding process, at least 41 different codes were considered. Through a process of attrition and conglomeration, 10 codes emerged. Table 2 lists the codes and keywords used in this process.
Table 2 Codes and keywords used in analysis Three of the intrinsic motivation items invoked fun and enjoyment, and two of the interviewees explicitly referenced these terms. As Bill stated, “It’s enjoyable because of the complexity”, and Ted said, “Some stuff that is challenging can be enjoyable”. These two students also commented, “It’s fun to learn new things”, and “If something is fun, you enjoy doing it”. The intrinsic motivation item describing mathematics as interesting led interviewees to reflect on engagement, mastery, and relevance. This was often stated in a negative light. Louise said, “[Mathematics] is just one way of doing something”. When questioned for clarity, she stated, “Interesting would be something where you could be creative?” Although not an endorsement, the concept of being creative may have implied autonomy, and this implied cognitive processes that were associated with intrinsic motivation.
Some of the responses to the mastery orientation items concerned wanting to improve. Thelma states she “would want to learn new math concepts because [she] wants to progress mathematically”, or as Bill states, “It’s always good to learn new things and I would like to master some of those things”. One mastery item invoked mastery avoidance processes. As Ted states, “I wouldn’t want to have a mathematician mindset for understanding mathematics”. Whether the items invoked a desire to understand mathematics or a desire to avoid understanding, these responses provided evidence of cognitive processes related to mastery orientation.
Similarly, there was evidence that the performance-oriented items invoked constructs associated with performance orientation. When discussing doing better than other students in the class, Thelma stated, “It's always nice to say I was in the group that got the A's”. This social connotation associated with performance orientation and being part of a group appeared in many of the comments concerning performance; however, the students in this study may have social norms that are unique to this population. As Bill stated, “I don’t really focus on what the other students are doing … but the grade will ultimately reflect who’s been doing better”. One of the items under review included the phrase looking smart. These same respondents took issue with this term. As Louise stated, “I don’t like to look smart … [but to] be as smart as other people think I look, that would be nice”. For these students, fitting in was important, and looking smart and being better than others may have had negative connotations. Although, social comparisons were generally viewed negatively, these responses did lend evidence that the items referenced cognitive processes associated with performance orientation.
Items associated with expectancy generally invoked cognitive processes associated with self-efficacy and expectations of success. As Thelma stated, “Right now I’m doing very well in my math class … doing all my homework, studying, putting in time and effort … [and] if I keep that up I know that I will get a good grade”, and as Ted stated, “[If] I do put all my active effort into it, I definitely will [master the concepts]”. In general, responses to the expectancy items referenced cognitive processes associated with expectancy.
To verify content, we computed effect sizes defined as the proportion of concept phrases in our intra-respondent matrix that exemplify a code (Onwuegbuzie 2003). See Table 3 for the effect size per code for each item contained in the final MMAI. This revealed that the intrinsic motivation items tended to provoke cognitive processes associated with fun, efficacy, engagement, and affect. The mastery-oriented items evoked cognitive processes associated with relevance and mastery of content. The performance-oriented items evoked cognitive responses associated with performance comparisons and relationships. The expectancy items were associated with students’ perceived efficacy surrounding their classroom experiences.
Table 3 Summary of intra-respondent matrix Based on analyses of these interviews, we removed 15 items that did not reveal evidence of being representative of their intended construct. In this way, using an evidence-based approach, we narrowed in on four items each to represent intrinsic motivation, performance orientation, mastery orientation, self-efficacy, and expectancy, and three items each to represent task value and extrinsic motivation. Table 4 lists the reasons for the removal of the intrinsic motivation, performance orientation, mastery orientation, and expectancy items.
Table 4 Evidence supporting removal of items from the four-factor Motivation for Mathematics Abbreviated Instrument (MMAI) Phase 4: preliminary instrument
IRT for preliminary instrument
To begin an analysis using IRT, we administered this 26-item survey to 186 undergraduate college students taking their first math course at a public university in the southern part of the United States. Assuming all these items were measuring motivation, we compared a model with the item loadings constrained to be equal to a model where the item loadings were not constrained. The difference in the chi-square fit indices was significant \({\rm X}^{2}\)(25) = 111.765, p < 0.0001 implying the item loadings were significantly different. We analysed the item information curves for the unconstrained model, as well as the item characteristic curves. We found the information curves for the items representing self-efficacy, extrinsic motivation, and performance orientation were flat, providing little information for measuring the overall construct. The item characteristic curves for these poor performing items also revealed smooth low overlapping hills revealing no delineation between answers. See Fig. 1 for the item information curves both with and without inclusion of the poor performing items.
EFA for preliminary instrument
We then conducted an EFA of the preliminary instrument. A plot of initial eigenvalues revealed a steepening of the slope between the fourth and fifth eigenvalue, and parallel analysis revealed the fourth eigenvalue for the actual data above and the fifth eigenvalue below the eigenvalues from the simulated random data. Thus, the data are best fit with a four-factor model. See Fig. 2 for eigenvalue plot.
Analysis of the rotated pattern matrix revealed the intrinsic motivation, mastery orientation, performance orientation, and expectancy items separating cleanly into the four factors where within factor loadings were all greater than 0.74 and all loadings across factors being less than 0.16. The extrinsic motivation and self-efficacy items loaded across factors, and therefore we removed them from the instrument. We also removed the task-value items as they weakly loaded on one factor. See Table 5 for the pattern matrix. The items representing intrinsic motivation, mastery orientation, and expectancy performed best under the IRT analysis, and these items as well as the items representing performance orientation revealed a strong four-factor structure. As a result of theoretical concerns and these statistical results, we focussed the MMAI on a four-factor model with the four factors being intrinsic motivation, mastery orientation, performance orientation, and expectancy. Table 6 lists the items contained in the final 16-item four-factor MMAI.
Table 5 Geomin rotated pattern matrix for exploratory factor analysis of preliminary data Table 6 Items contained in the four-factor MMAI Phase 5: internal structure of the MMAI
We administered the 16-item MMAI to 386 university participants in general education mathematics courses offered from the fall of 2015 through the summer of 2017 at a public university in the southern United States. Of these participants, 211 identified as females, 167 as males, and eight students identified as other or choose not to answer.
CFA for final instrument
We conducted a CFA of the four-factor 16-item MMAI, and although the chi-square value of \({\rm X}^{2}\)(98, n = 386) = 212.02, p < 0.0001 indicated misfit, other fit indices less sensitive to sample size revealed a good fit as per levels reported by Browne and Cudeck (1993), MacCallum et al. (1996), and Hu and Bentler (1999). The four-factor model had an RMSEA = 0.055. The SRMR was 0.044, and the CFI was 0.963. All indicate a good fit.
All other sources of misfit with modification indices greater than 10 concerned correlated errors of items within factors. Some expectancy items had correlated errors. Ex3 with Ex1 had a modification index of \({\rm X}^{2}\)(1) = 12.70, p < 0.0001. Ex4 with Ex2 had a modification index of \({\rm X}^{2}\)(1) = 16.12, p < 0.0001. Some mastery items had correlated errors for Mo2 with Mo1 and Mo4 with Mo1. Their modification indices were \({\rm X}^{2}\)(1) = 14.39, p < 0.0001 and \({\rm X}^{2}\)(1) = 10.43, p < 0.0001, respectively. Two performance orientation items—Po1 and Po3—also had correlated error with a modification index of \({\rm X}^{2}\)(1) = 12.62, p < 0.0001. Modification indices showed some correlated error between items within the same latent factors, however, in view of the good fit of the original model, none of these sources of misfit warranted changing the model. See Table 7 for fit indices and modification indices.
Table 7 Fit indices and modification indices for CFA Internal reliability and validity
To examine the internal reliability of the measurement model, we found the composite reliability for all four latent factors to be excellent with a lowest of 0.86 and the highest at 0.93. These are well above Fornell and Larcker’s (1981) minimum threshold of 0.70. To examine convergent validity in the measurement model, we compute AVE for each latent construct. These values were also excellent as the lowest of 0.60 was well above Fornell and Larcker’s (1981) minimum threshold of 0.50. To examine discriminate validity in the measurement model, we compared the squared correlations between factor loadings to AVE with the requirement that \({\uprho }_{\text{AVE}({\upxi }_{j})}\)>\({\gamma }^{2}\) , where \(\gamma\) is the correlation of the factor \({\upxi }_{j}\) with any other factor. All four factors met this requirement. All individual item reliabilities were above Ab Hamid et al.’s (2017) minimum threshold of 0.4, however, the first performance orientation item was concerning with \({\uprho }_{\text{po}1}=0.44\). See Table 8 for the complete summary.
Table 8 Summary table for reliability and validity estimates for the measurement model Significant relationships
All item loadings on the latent factors were significantly different from zero with p < 0.001. The range of the standardized loadings for the intrinsic factor was 0.80 to 0.93, the range for the mastery orientation factor was 0.78 to 0.83, the range for the performance orientation factor was 0.66 to 0.83, and the range for the expectancy factor was 0.76 to 0.86. All correlations between latent factors were also significantly different than zero with p < 0.001. Correlations between the factors ranged from 0.22 (performance and mastery orientation) to 0.49 (intrinsic and mastery orientation). These low-factor inter-correlations provided evidence of discriminant validity between factors. See Fig. 3 for the CFA model.
Measurement invariance
We conducted invariance testing grouping students as male and female. Starting with a configural model, we successively added constraints and compared the successive fit indices. For the metric model, we constrained the factor loading to be the same across groups. For the scalar model we also constrained the intercepts to be the same, and finally, for the strict model, both item level and latent factor level errors were constrained to be the same. Using the Satorra and Bentler (2010) scaled chi-square differences, we only found significant change in fit between the metric and scalar models (\({\rm X}^{2}\)(12) = 26.35, p = 0.01). RMSEA stayed remarkably stable fluctuating between 0.058 and 0.059 throughout this entire process, however, CFI and SRMR showed some degradation from the added constraints with CFI moving from 0.958 to 0.951 and SRMR moving from 0.053 to 0.071. See Table 9 for a summary of the invariance testing data.
Table 9 Summary table for measurement invariance