The Development of the STEM Career Interest Survey (STEM-CIS)
- First Online:
- Cite this article as:
- Kier, M.W., Blanchard, M.R., Osborne, J.W. et al. Res Sci Educ (2014) 44: 461. doi:10.1007/s11165-013-9389-3
Internationally, efforts to increase student interest in science, technology, engineering, and mathematics (STEM) careers have been on the rise. It is often the goal of such efforts that increased interest in STEM careers should stimulate economic growth and enhance innovation. Scientific and educational organizations recommend that efforts to interest students in STEM majors and careers begin at the middle school level, a time when students are developing their own interests and recognizing their academic strengths. These factors have led scholars to call for instruments that effectively measure interest in STEM classes and careers, particularly for middle school students. In response, we leveraged the social cognitive career theory to develop a survey with subscales in science, technology, engineering, and mathematics. In this manuscript, we detail the six stages of development of the STEM Career Interest Survey. To investigate the instrument's reliability and psychometric properties, we administered this 44-item survey to over 1,000 middle school students (grades 6–8) who primarily were in rural, high-poverty districts in the southeastern USA. Confirmatory factor analyses indicate that the STEM-CIS is a strong, single factor instrument and also has four strong, discipline-specific subscales, which allow for the science, technology, engineering, and mathematics subscales to be administered separately or in combination. This instrument should prove helpful in research, evaluation, and professional development to measure STEM career interest in secondary level students.
KeywordsSTEM interestInstrumentSurveySocial cognitive career theorySTEM careersConfirmatory factor analysis
Many countries around the world face the task of recruiting more individuals into science, technology, engineering, and mathematics (STEM) industries (Hill et al. 2010; Regisford 2012). Countries such as Austria, France, Germany, Honduras, Mexico, The Netherlands, and Switzerland struggled during the recent economic recovery with few individuals trained in using and creating the technologies capable of improving domestic production (Schwab and Sala-i-Martín 2012). Similarly, in the USA, labor statistics project the development of nearly three million new jobs in science and technology by 2020, requiring capable individuals with educational backgrounds in STEM to fill these positions (US Bureau of Labor Statistics 2010; National Science Board 2010). With only 16 % of US students (most of whom are men; Cataldi et al. 2011) obtaining a STEM-related degree out of 1.6 million bachelor's degrees, leaders are concerned with the lack of creativity and perspectives from a limited labor pool (NSF 2009).
Policy leaders around the world are creating partnerships between large businesses organizations and teachers to recruit and engage more students in STEM areas, expanding career opportunities for females and minorities, and creating future STEM innovations (e.g., Change the Equation 2010; Regisford 2012; Tech Women 2013; White House Office of Science and Technology Policy 2012). A European national report suggests that students are becoming less willing to pursue a STEM field in college, with STEM graduates falling from 24.8 % in 1999 to 22.7 % in 2005 and continuing to decline (Business Europe 2011). Labor statistics examining Australian applicants for STEM-intensive jobs show comparatively fewer qualified applicants for engineering jobs compared to those applicants in natural sciences and medical fields. For example, approximately 43 people apply for each job vacancy in petroleum engineering, and on average, none of them are suitably qualified (Healy et al. 2011). Thirty percent of Australian employers in information technology also report difficulties in recruiting qualified applicants in vacant positions (Healy et al.). The USA bears a similar burden of not being able to find qualified individuals to fill nearly 600,000 jobs in the manufacturing sector, with difficulties finding qualified individuals for technical jobs and jobs in engineering (STEMconnector 2012).
A variety of reports suggest reasons why students may hesitate to pursue STEM courses and careers, including a lack of quality preparation in mathematics and science in K-12 educational systems, lack of access to money and technology, lack of guidance from adults who are knowledgeable of or affiliated with STEM careers, psychological barriers (such as believing mathematics and science are too difficult) and lack of role models in the fields (Drew 2011; National Academy of Sciences, Global Affairs & Institute of Medicine 2011; Scott and Martin 2012). Studies also document a general decline in STEM interest from elementary school to late high school (VanLeuvan 2004; Wells et al. 2007), but few studies have measured why student interest in STEM subjects or careers changes (and often declines) prior to entering college (Fouad et al. 1997; VanLeuvan 2004). Educational testing organizations and researchers suggest that teachers promote STEM careers in the classroom, beginning at the middle school level (e.g., American College Testing 2011; Skamp 2007). When elementary and middle school students are engaged in discussions about goals and opportunities available in STEM, they have time to connect their interests to these subjects, and demonstrate higher self-efficacy in these fields prior to college (Skamp 2007).
Studies that have used in-school and out-of-school interventions designed to connect underrepresented students to STEM professionals and careers show promise in increasing awareness and interest in STEM careers (Avery 2013; Blanchard et al. 2012). Specifically, interventions that include STEM role models have been found very effective in increasing students' engagement and creating accurate perceptions of STEM with students at the secondary and postsecondary level (Ashby Plant et al. 2009; Stout et al. 2011; Zeldin et al. 2008). When examining the literature in counselor education and STEM education, few instruments have been developed to measure the construct of interest in STEM careers in general, and specifically at the middle school level (Whitfield et al. 2008), an age when students are forming career beliefs (ACT 2011; Skamp 2007).
Tyler-Wood et al. (2010) address this gap with two instruments, the STEM Semantics Survey and the STEM Career Questionnaire. The researchers validated the STEM Semantics Survey with a population that ranged in age from middle school to adult. Their STEM Semantics Survey contains five pairs of opposing adjectives within each subject area and for STEM careers as a whole. For example, students rank on a 7-point Likert scale if they feel that science is better described as fascinating (1) or mundane (7). They calculated Cronbach's alphas for science, technology, engineering, mathematics, and general STEM careers that ranged from .84 to .93. They suggest using this survey in combination with validated constructs from Bowdich's (2009) Career Interest Questionnaire (CIQ). The CIQ constructs adapted by Tyler-Wood et al. consisted of 12 items that measure perception of a supportive environment for pursuing a career in science, interest in pursuing educational opportunities that would lead to a career in science, and perceived importance of a career in science. Cronbach's alphas were established for 60 middle school students and ranged from 0.78 to 0.94. The authors' use of exploratory factor analysis on both surveys provides strong implications that these items are effective in measuring interest in STEM. However, neither survey was explicitly linked to any theoretical framework.
Lent et al. (1994) developed a promising theoretical model to predict interest and intent to pursue academic choices and careers, the social cognitive career theory (SCCT). This model allows researchers to use measures of individual's self-efficacy, outcome expectations, personal inputs and backgrounds, and contextual supports and/or barriers to explain reasoning behind students' academic or career choices. Studies that have utilized this theory as a predictive model of interest for STEM fields have predominately been at the secondary and postsecondary level. However, as no STEM career interest survey existed, these researchers used a variety of surveys that measured one or more aspect of the SCCT, such as a survey on the nature of learning experiences in mathematics class (Hill 2011), self-efficacy in science and mathematics (Fouad et al. 1997), and intent to pursue careers in information technology (Stone et al. 2005). Another method for incorporating aspects of the SCCT to predict student interest in science and mathematics has imputed school-based data into the model (e.g., Navarro et al. 2007).
Navarro et al. (2007) investigated science and mathematics interest in Mexican American middle school students using generation status, socioeconomic data from the district, and separate surveys on outcome expectations and self-efficacy from the students. Whitfield et al. (2008) have identified ten instruments to measure career interests, but four of these use normative data, and none of them focus on STEM careers. No study to date has used the social cognitive aspects to develop a survey measuring interest and intent to pursue academic choices or careers in STEM for middle school students. Therefore, given the utility of this theoretical framework, we have developed a survey measuring interest in each subject area with aspects derived from the SCCT. We believe the STEM Career Interest Survey (STEM-CIS) instrument will help researchers in professional development and program evaluators to measure the effects of their STEM programs on changes in student interest in STEM subjects and careers, with implications for STEM intervention design and changes.
Studies that report findings on what influences STEM career interest at the middle and secondary levels guided the development of our survey items, in addition to our theoretical framework. There are fewer studies reporting findings on the middle school students' interest in STEM compared to what is known about secondary and postsecondary students (Usher 2009). One study that surveyed 2,500 middle school and high school students on their perceptions of the nature of science and scientists and on their interest in science and being a scientist found most students to be very interested in science but have little interest in becoming a scientist (Kitts 2009).
One reason that students may not see themselves in STEM careers is because of a perception that these careers are too difficult and require too much education (American Association of State Colleges and Universities 2005; Drew 2011). The importance of self-efficacy in developing career interests and forming academic goals has been well supported by theory-based models such as the Social Cognitive Career of Lent et al. (1994) and by Eccles' (1994) model of achievement-related choices. Research findings also report that STEM interest may differ between male and female students, with high school male students reporting more interest in physical sciences and females in the biological sciences (Baram–Tsabari et al. 2009). Frome et al. (2006) suggest that societal roles and gender play a role in students' self-efficacy and interest. Brotman and Moore (2008) found that eight grade female students identified fewer informal experiences with science than did male students, another limiting factor to why female students in particular may be less confident and in turn less interested in the physical science and engineering fields. Similar to Brotman and Moore, Archer et al. (2010) conducted focus groups with middle school students and also found girls had fewer experiences than boys and associated science with being dangerous. In addition, students from more affluent neighborhoods experienced more parent-guided informal science experiences compared to students in neighborhoods of lower SES.
Studies that address the perceptions of K-12 students regarding their perceptions of STEM careers and professionals suggest that students have little experience with these careers (Masnick et al. 2010). Inaccurate stereotypes of STEM professionals begin at an early age; one study found that elementary students commonly drew and described engineers as men, and someone who fixes things similar to a mechanic (Capobianco et al. 2011). The trend continues in the secondary years with students describing engineers as males who work on cars, fix or build things, work on trains, repair electronics, or generally design (Capobianco et al. 2011; Fralick et al. 2009). Similar to scientists and engineers, professionals in information technology (IT) careers are stereotyped as noncollaborative and monotonous by postsecondary students (Thomas and Allen 2006). Johnson and Miller (2002) assert that these careers are likely not attractive to students, particularly female students, because of the manner in which they are advertised (e.g., a men-only environment, long hours that would not be conducive to a family). They encourage that these careers in particular need to be more clearly articulated to school students to attract more diverse individuals.
Lent et al. (1994, 2000) provide a promising theoretical framework and predictive model on interest and career choice, the social cognitive career theory (SCCT). The SCCT is based on Bandura's (1986) social cognitive theory of learning. According to Bandura, the most influential component to goal setting and action is self-efficacy, an individual's belief that he or she is capable of mastering events within his or her life. For example, a student with high science self-efficacy may believe that she is able to earn a good grade in her science class. Self-efficacy is involved in setting personal goals, analyzing decisions, and making commitments.
The Current Study
Our goal was to develop the STEM-CIS based on key aspects of the SCCT (e.g., self-efficacy, outcome expectations, personal inputs, and contextual supports and barriers). Below, we discuss the stages of development of the STEM-CIS, as well as how this new instrument functioned in measuring factors related to middle school students' interest in and goals related to STEM subjects and potential careers.
- Stage 1
Conduct a literature review to help develop relevant items
- Stage 2
Create a broad item pool of items that will test the target aspect
- Stage 3
Preliminary pilot testing of items
- Stage 4
Conduct structural analysis to determine which items are to be eliminated from the pool of items
- Stage 5
Perform a factor analysis
- Stage 6
Demographics of participant schools
Free and reduced lunch
Participants in the Current Study
Data used in this validation process were collected from 1,061 students at seven middle schools in the rural southeastern USA (grades 5–8; see Table 1 for school demographics). For all students, it was the first time they had the seen the items. Students took the survey online using either an iPod touch® or a computer.
Stage 1: Literature Review
The literature review consisted of a search for studies addressing students' interest in STEM and STEM careers, students' perceptions of STEM professionals, and social cognitive career theory applied in mathematics and science settings. The search included using ERIC, JSTOR, and Google Scholar and searching under the terms student STEM interest, instruments measuring STEM, students' perceptions of STEM, and social cognitive career theory and STEM, from the past 10 years. Individual disciplines within STEM were also searched to find literature regarding interest, perceptions of careers, measurements of interest, and social cognitive predictors of interest. Research from approximately 130 articles was accumulated to write a larger review of STEM education (Kier 2013), with only select references included in this manuscript. As previously described, findings suggested that interest in careers was often related to self-efficacy, outcome expectations, and previous learning experiences—addressed by Lent et al.'s social cognitive career theory. The literature review and theoretical framework guided the development of our initial pool of survey items, as well as other instruments measuring STEM courses and careers (e.g., Tyler-Wood et al. 2010). The literature indicated the need for a survey instrument that was developed for middle school students (Tyler-Wood et al. 2010; Usher 2009).
Stage 2: Create a Broad Item Pool of Items That Will Test the Target Aspect
Initial survey items were developed that connected to each of the aspects of the social cognitive career theory. We were guided by our literature review, a middle school mathematics and science self-efficacy scale (Fouad et al. 1997), and eight separate classroom STEM career discussions with sixth–eighth grade students at one of the schools participating in our development of the initial survey items with age-appropriate wording.
Examples from a previous version of the survey
Original questions numbers
SCCT aspects measured
Original Cronbach's alpha
I am able to get a good grade in science
I intend to enter a career that uses science
If I do well in my mathematics classes, it will help me in my future career
I am interested in careers that use science
I would feel comfortable talking to people who work in mathematics careers
Contextual supports and barriers
I have people in my life that would help me get to a career in mathematics or science
Stage 3: Preliminary Pilot Testing of Items
These 30 survey items developed in stage 2 were piloted with 61 students in a diverse urban middle school that was not part of the final sample. Students completed the online survey using an iPod touch®, which took approximately 10 minutes.
Structural analyses from this pilot study found poor correlation scores between items and lower than anticipated alphas within each SCCT aspect. Several questions combined mathematics and science into a single item (e.g., It is too hard to get a career in mathematics and science; If I get good grades in mathematics and science, my parents will approve of me) and thus were eliminated or reworded due to the ambiguity as to which content area (mathematics or science) a student might have been responding.
Demographics of pilot schools
Free and reduced lunch
Stage 4: Conducting Structural Analysis to Determine Which Items Are to be Eliminated from the Pool of Items
Item correlations for STEM-CIS
Alpha or item–total correlation
Social cognitive career theory aspect
Science α = 0.77, average scale score = 27.61, SD = 7.19, N = 831
I am able to get a good grade in my science class
I am able to complete my science homework
I plan to use science in my future career
I will work hard in my science classes
If I do well in science classes, it will help me in my future career
My parents would like it if I choose a science career
Interest in science
I am interested in careers that use science
Interest in science
I like my science class
I have a role model in a science career
I would feel comfortable talking to people who work in science careers
I know of someone in my family who uses science in their career
Mathematics α = 0.85, average scale score = 25.17, SD = 8.52, N = 829
I am able to get a good grade in my mathematics class
I am able to complete my mathematics homework
I plan to use mathematics in my future career
I will work hard in my mathematics classes
If I do well in mathematics classes, it will help me in my future career
My parents would like it if I choose a mathematics career
Interest in mathematics
I am interested in careers that use mathematics
Interest in mathematics
I like my mathematics class
I have a role model in a mathematics career
I would feel comfortable talking to people who work in mathematics careers
I know someone in my family who uses mathematics in their career
Technology α = 0.89, average scale score = 26.27, SD = 10.41, N = 102
I am able to do well in activities that involve technology
I am able to learn new technologies
I plan to use technology in my future career
I will learn about new technologies that will help me with school
If I learn a lot about technology, I will be able to do lots of different types of careers
When I use technology in school, I am able to get better grades
Interest in technology
I like to use technology for class work
Interest in technology
I am interested in careers that use technology
I have a role model who uses technology in their career
I would feel comfortable talking to people who work in technology careers
I know of someone in my family who uses technology in their career
Engineering: α = 0.86, average scale score = 30.03, SD = 8.85 , N = 250
I am able to do well in activities that involve engineering
I am able to complete activities that involve engineering
I plan to use engineering in my future career
I will work hard on activities at school that involve engineering
If I learn a lot about engineering, I will be able to do lots of different types of careers
My parents would like it if I choose an engineering career
Interest in engineering
I am interested in careers that involve engineering
Interest in engineering
I like activities that involve engineering
I have a role model in an engineering career
I would feel comfortable talking to people who are engineers
I know of someone in my family who is an engineer
Summary of confirmatory factor analyses
Science only (N = 1061)
Mathematics only (N = 1061)
Technology only (N = 1056)
Engineering only (N = 1061)
All four factors (N = 1061)
All items on one factor (N = 1061)
Stages 5 and 6: Confirmatory Factor Analysis of Subscales
To verify the psychometric properties of each subscale (i.e., whether each subscale was one-dimensional with desirable model fit), each content scale (science, mathematics, technology, and engineering) was subjected to confirmatory factor analysis via AMOS.1 For each subscale analysis, a single-factor solution was modeled using data from cases with complete data on those subscale items. Modification indices were examined to optimize the fit of the initial model.2 This resulted in slightly different parameters estimated for each content scale. Once a model was specified appropriately, the full data were imported into the model. AMOS can utilize cases with partial information (i.e., it does not require complete data for all cases and, as such, does not delete cases with partial missing data from the analysis) and all CFA analyses reported below are therefore based on the full sample of N = 1,061.
For the science subscale analyses, Mahalanobis D2 was used to evaluate multivariate outliers.3 In this scale, a reasonable cutoff for D2 with 38 parameters estimated would have been 53.38. No outliers with a D2 over 36 were observed. Testing model fit in confirmatory factor analyses (CFAs) routinely involves evaluating and reporting multiple fit indices, as each one captures different aspects of model fit. The chi square statistic is the first measure of how well a model fits the data; however, it is greatly influenced by sample size and thus is rarely found to be nonsignificant in samples sufficiently large enough to legitimately perform CFA within (Thompson and Daniel 1996).
Therefore, an appropriate model fit was assessed using the root mean square error (RMSEA), the comparative fit index (CFI), and the normed fit index (NFI). Model fit is usually considered adequate when the following criteria are met: RMSEA should be <0.08, NFI should meet or exceed 0.90, and CFI should meet or exceed 0.954 (Byrne 2010; Hu and Bentler 1999; Marsh et al. 2004). Model chi squared generally only is used for comparing nested models, as it is determined both by sample size and goodness of fit, and CMIN/df is a commonly reported scaling of chi square, divided by the degrees of freedom in the model. In general, for both of these indices, smaller numbers are better, but there is no widely accepted way to interpret them objectively.
As you can see in Table 5, model fit for this scale was strong, exceeding all recommendations for NFI, CFI, and RMSEA, indicating that this subscale indeed represents a single coherent factor. Further, as Fig. 3 shows, standardized factor loadings were generally strong, ranging from 0.29 to 0.70.
The STEM Career Interest Survey developed in this study was shown to be psychometrically sound for each of the subscales of science, technology, engineering, and mathematics. Because some researchers will be interested in student interest toward more than one of these content areas, we also explored how these four subscales worked together as a single measure. This is a reasonable approach as analyses showed the four latent subscale scores to be strongly correlated; correlations between latent factors ranged from r = .72 to .82. Thus, we estimated two other models: all four subscales in a single model as four separate but correlated factors and all items from all four subscales characterized as one single factor (Appendix).
Because of the complex sampling history of this project, large amounts of imputation would be undesirable. Thus, these models were estimated using all 1,061 students but without imputing missing data. This has the desirable effect of not relying upon hundreds of imputed engineering scale scores, for example, but has the undesirable side effect of not allowing the model to be fit using modification indices, and thus, the model fit for these last two models is probably lower than otherwise might be achieved. We also were not able to examine the data for outliers, again, which could harm model fit substantially.
Table 5 shows reasonable model fit for the four-factor model. In particular, RMSEA indicates strong model fit, despite NFI and CFI falling slightly below norms for strong models [recall Marsh et al. (2004) argues that strict adherence to cutoff rules is not defensible]. Thus, it is our conclusion that it is defensible to use this instrument in middle grade populations with all four subscales characterized by four separate subscores.
For researchers who seek one single overall score from these four subscales (or from several of them), the evidence from the single-factor model that included all items from all four scales loading on a single factor shows reasonable model fit, with a RMSEA well below norms and CFI and NFI that are reasonably close to established guideposts for acceptable model fit. This model fit is also significantly better than the four-subscale model (ΔΧ2 = 389.22, df = 70, p < .000000000001; ΔCFI = 0.03, ΔNFI = 0.04, and ΔRMSEA = 0.005). This final analysis supports combining all items into a single overarching scale score, should researchers find that desirable (see Table 5).
Thus, it is possible to use this scale in a variety of ways, each defensible and supported by these analyses. It is defensible to use any of the four subscales individually, to use more than one (or even all four) together in a single instrument, calculating individual subscale scores for each content area, and it is also defensible to use more than one subscale and combine all items into a single overall score reflecting career interests in the STEM field overall.
Limitations of the Current Study
There are myriad limitations to any study as ambitious as this one. The first and most obvious limitation involves the sample. Any psychometric property of a scale is contingent on the sample—in other words, your results might differ from ours using the same scale and a different sample. In future work, we expect to evaluate the stability of the model using cross-validation and invariance analyses.
Another limitation is that we have not reported any analyses that speak to whether this scale is equally appropriate for both girls and boys (or for different racial/ethnic or age groups). It is reasonable to wonder, for example, whether girls and boys have equally strong internal consistency coefficients, or identical factor structures. These types of analyses, invariance analyses, are lengthy and complex to report, and thus, for the sake of brevity, we have chosen not to report them in this paper. They also ideally require much larger and more diverse samples, which we are currently engaged in collecting. Initial analyses (not reported here) indicate strong invariance across gender groups. We anticipate reporting these results more fully in a subsequent paper.
Finally, as has been noted above, our sampling framework was somewhat complex and driven by the processes involved in the execution of the grant that funded the data collection and the realities of data collection with real students in high-poverty school districts.
The STEM-CIS was developed to measure the effects of strategies intended to promote the awareness of, interest in, and intent to pursue STEM careers with rural, minority, middle school students (Blanchard et al. 2012). Its development thus addresses calls for an age appropriate measure (Gushue 2006; Lent et al. 2008) that is theoretically strong (Whitfield et al. 2008). With this particular population, we were interested in finding out not only if students were interested in STEM subjects and careers but also what factors influenced this interest.
The development of this survey built on previous instruments (Fouad et al. 1997; Tyler-Wood et al. 2010), as well as a promising framework, the social cognitive career theory (Lent et al. 1994, 2000). The SCCT has been used and psychometrically evaluated in predicting interest with middle school students and now has been applied to this new STEM career interest survey.
The STEM Career Interest Survey developed in this study was shown to be psychometrically sound and able to be used by researchers or professional developers in science, technology, engineering, and mathematics, using one or more subscales or all of them as one instrument, as needed. As such, we expect it will be beneficial to researchers, professional developers, and evaluators in measuring STEM career interest and the effects of STEM programs on changes in student interest in STEM subjects and careers. The knowledge that we gain from the use of this instrument may help to inform efforts taken at the middle school level as we seek to increase students' interest in STEM subjects, majors, and careers. We also plan to validate this instrument with other populations of students in urban settings and at the high school level. Innovations worldwide depend on highly qualified professionals in STEM careers (Hill et al. 2010; Regisford 2012; White House Office of Science and Technology Policy 2012). We hope that the STEM-CIS will make a contribution to studying the impact of our efforts to attract students to meet these needs.
AMOS is the structural equation modeling software produced with IBM/SPSS.
Modification indices are reported in AMOS as chi square values with one degree of freedom. As such, a significant improvement in the model is signified by a chi square in excess of 3.84. In order to be very conservative with this process, and to model inter-item correlations that contributed substantially to model fit, we used a rule that we would examine inter-item correlations that resulted in a modification index (MI) of 8.00 or greater. Further, note that because these models were all single latent variable models with multiple measured variables that legitimately should be allowed to correlate, there is no conceptual issue with allowing items designed to measure a particular latent variable to correlate. We cannot model all possible correlations between items on a scale in CFA due to lack of degrees of freedom, and thus, the use of modification indices is merely an efficient method of identifying those items that, by allowing a nonzero correlation, significantly and substantially (and appropriately) improve model fit.
Mahalanobis D2 is a common index of whether an individual score is aberrant within the multivariate distribution of scores; higher numbers indicate the score is farther from the center of the multivariate distribution. Guidelines for assessing Mahalanobis D2 suggest that a reasonable cutoff score is a chi square value that would be significant at p < .05 for the number of df equal to the number of variables or parameters estimated in the model.
The authors wish to thank Michael D. Cobb for his helpful suggestions on the initial development of this instrument and all of the collaborators on this project who participated in this research. This research was funded by an ITEST grant (2010–2013) from the National Science Foundation (award number 1031118). The opinions expressed are those of the authors and do not represent the views of the National Science Foundation or North Carolina State University.