Research in Science Education

, Volume 44, Issue 3, pp 461–481

The Development of the STEM Career Interest Survey (STEM-CIS)

Authors

    • Department of Curriculum and InstructionHoward University
  • Margaret R. Blanchard
    • Department of Science, Technology, Engineering, and Mathematics EducationNorth Carolina State University
  • Jason W. Osborne
    • College of Education & Human DevelopmentUniversity of Louisville
  • Jennifer L. Albert
    • Department of Computer ScienceNorth Carolina State University
Article

DOI: 10.1007/s11165-013-9389-3

Cite this article as:
Kier, M.W., Blanchard, M.R., Osborne, J.W. et al. Res Sci Educ (2014) 44: 461. doi:10.1007/s11165-013-9389-3

Abstract

Internationally, efforts to increase student interest in science, technology, engineering, and mathematics (STEM) careers have been on the rise. It is often the goal of such efforts that increased interest in STEM careers should stimulate economic growth and enhance innovation. Scientific and educational organizations recommend that efforts to interest students in STEM majors and careers begin at the middle school level, a time when students are developing their own interests and recognizing their academic strengths. These factors have led scholars to call for instruments that effectively measure interest in STEM classes and careers, particularly for middle school students. In response, we leveraged the social cognitive career theory to develop a survey with subscales in science, technology, engineering, and mathematics. In this manuscript, we detail the six stages of development of the STEM Career Interest Survey. To investigate the instrument's reliability and psychometric properties, we administered this 44-item survey to over 1,000 middle school students (grades 6–8) who primarily were in rural, high-poverty districts in the southeastern USA. Confirmatory factor analyses indicate that the STEM-CIS is a strong, single factor instrument and also has four strong, discipline-specific subscales, which allow for the science, technology, engineering, and mathematics subscales to be administered separately or in combination. This instrument should prove helpful in research, evaluation, and professional development to measure STEM career interest in secondary level students.

Keywords

STEM interestInstrumentSurveySocial cognitive career theorySTEM careersConfirmatory factor analysis

Many countries around the world face the task of recruiting more individuals into science, technology, engineering, and mathematics (STEM) industries (Hill et al. 2010; Regisford 2012). Countries such as Austria, France, Germany, Honduras, Mexico, The Netherlands, and Switzerland struggled during the recent economic recovery with few individuals trained in using and creating the technologies capable of improving domestic production (Schwab and Sala-i-Martín 2012). Similarly, in the USA, labor statistics project the development of nearly three million new jobs in science and technology by 2020, requiring capable individuals with educational backgrounds in STEM to fill these positions (US Bureau of Labor Statistics 2010; National Science Board 2010). With only 16 % of US students (most of whom are men; Cataldi et al. 2011) obtaining a STEM-related degree out of 1.6 million bachelor's degrees, leaders are concerned with the lack of creativity and perspectives from a limited labor pool (NSF 2009).

Policy leaders around the world are creating partnerships between large businesses organizations and teachers to recruit and engage more students in STEM areas, expanding career opportunities for females and minorities, and creating future STEM innovations (e.g., Change the Equation 2010; Regisford 2012; Tech Women 2013; White House Office of Science and Technology Policy 2012). A European national report suggests that students are becoming less willing to pursue a STEM field in college, with STEM graduates falling from 24.8 % in 1999 to 22.7 % in 2005 and continuing to decline (Business Europe 2011). Labor statistics examining Australian applicants for STEM-intensive jobs show comparatively fewer qualified applicants for engineering jobs compared to those applicants in natural sciences and medical fields. For example, approximately 43 people apply for each job vacancy in petroleum engineering, and on average, none of them are suitably qualified (Healy et al. 2011). Thirty percent of Australian employers in information technology also report difficulties in recruiting qualified applicants in vacant positions (Healy et al.). The USA bears a similar burden of not being able to find qualified individuals to fill nearly 600,000 jobs in the manufacturing sector, with difficulties finding qualified individuals for technical jobs and jobs in engineering (STEMconnector 2012).

Literature Review

A variety of reports suggest reasons why students may hesitate to pursue STEM courses and careers, including a lack of quality preparation in mathematics and science in K-12 educational systems, lack of access to money and technology, lack of guidance from adults who are knowledgeable of or affiliated with STEM careers, psychological barriers (such as believing mathematics and science are too difficult) and lack of role models in the fields (Drew 2011; National Academy of Sciences, Global Affairs & Institute of Medicine 2011; Scott and Martin 2012). Studies also document a general decline in STEM interest from elementary school to late high school (VanLeuvan 2004; Wells et al. 2007), but few studies have measured why student interest in STEM subjects or careers changes (and often declines) prior to entering college (Fouad et al. 1997; VanLeuvan 2004). Educational testing organizations and researchers suggest that teachers promote STEM careers in the classroom, beginning at the middle school level (e.g., American College Testing 2011; Skamp 2007). When elementary and middle school students are engaged in discussions about goals and opportunities available in STEM, they have time to connect their interests to these subjects, and demonstrate higher self-efficacy in these fields prior to college (Skamp 2007).

Studies that have used in-school and out-of-school interventions designed to connect underrepresented students to STEM professionals and careers show promise in increasing awareness and interest in STEM careers (Avery 2013; Blanchard et al. 2012). Specifically, interventions that include STEM role models have been found very effective in increasing students' engagement and creating accurate perceptions of STEM with students at the secondary and postsecondary level (Ashby Plant et al. 2009; Stout et al. 2011; Zeldin et al. 2008). When examining the literature in counselor education and STEM education, few instruments have been developed to measure the construct of interest in STEM careers in general, and specifically at the middle school level (Whitfield et al. 2008), an age when students are forming career beliefs (ACT 2011; Skamp 2007).

Tyler-Wood et al. (2010) address this gap with two instruments, the STEM Semantics Survey and the STEM Career Questionnaire. The researchers validated the STEM Semantics Survey with a population that ranged in age from middle school to adult. Their STEM Semantics Survey contains five pairs of opposing adjectives within each subject area and for STEM careers as a whole. For example, students rank on a 7-point Likert scale if they feel that science is better described as fascinating (1) or mundane (7). They calculated Cronbach's alphas for science, technology, engineering, mathematics, and general STEM careers that ranged from .84 to .93. They suggest using this survey in combination with validated constructs from Bowdich's (2009) Career Interest Questionnaire (CIQ). The CIQ constructs adapted by Tyler-Wood et al. consisted of 12 items that measure perception of a supportive environment for pursuing a career in science, interest in pursuing educational opportunities that would lead to a career in science, and perceived importance of a career in science. Cronbach's alphas were established for 60 middle school students and ranged from 0.78 to 0.94. The authors' use of exploratory factor analysis on both surveys provides strong implications that these items are effective in measuring interest in STEM. However, neither survey was explicitly linked to any theoretical framework.

Lent et al. (1994) developed a promising theoretical model to predict interest and intent to pursue academic choices and careers, the social cognitive career theory (SCCT). This model allows researchers to use measures of individual's self-efficacy, outcome expectations, personal inputs and backgrounds, and contextual supports and/or barriers to explain reasoning behind students' academic or career choices. Studies that have utilized this theory as a predictive model of interest for STEM fields have predominately been at the secondary and postsecondary level. However, as no STEM career interest survey existed, these researchers used a variety of surveys that measured one or more aspect of the SCCT, such as a survey on the nature of learning experiences in mathematics class (Hill 2011), self-efficacy in science and mathematics (Fouad et al. 1997), and intent to pursue careers in information technology (Stone et al. 2005). Another method for incorporating aspects of the SCCT to predict student interest in science and mathematics has imputed school-based data into the model (e.g., Navarro et al. 2007).

Navarro et al. (2007) investigated science and mathematics interest in Mexican American middle school students using generation status, socioeconomic data from the district, and separate surveys on outcome expectations and self-efficacy from the students. Whitfield et al. (2008) have identified ten instruments to measure career interests, but four of these use normative data, and none of them focus on STEM careers. No study to date has used the social cognitive aspects to develop a survey measuring interest and intent to pursue academic choices or careers in STEM for middle school students. Therefore, given the utility of this theoretical framework, we have developed a survey measuring interest in each subject area with aspects derived from the SCCT. We believe the STEM Career Interest Survey (STEM-CIS) instrument will help researchers in professional development and program evaluators to measure the effects of their STEM programs on changes in student interest in STEM subjects and careers, with implications for STEM intervention design and changes.

Studies that report findings on what influences STEM career interest at the middle and secondary levels guided the development of our survey items, in addition to our theoretical framework. There are fewer studies reporting findings on the middle school students' interest in STEM compared to what is known about secondary and postsecondary students (Usher 2009). One study that surveyed 2,500 middle school and high school students on their perceptions of the nature of science and scientists and on their interest in science and being a scientist found most students to be very interested in science but have little interest in becoming a scientist (Kitts 2009).

One reason that students may not see themselves in STEM careers is because of a perception that these careers are too difficult and require too much education (American Association of State Colleges and Universities 2005; Drew 2011). The importance of self-efficacy in developing career interests and forming academic goals has been well supported by theory-based models such as the Social Cognitive Career of Lent et al. (1994) and by Eccles' (1994) model of achievement-related choices. Research findings also report that STEM interest may differ between male and female students, with high school male students reporting more interest in physical sciences and females in the biological sciences (Baram–Tsabari et al. 2009). Frome et al. (2006) suggest that societal roles and gender play a role in students' self-efficacy and interest. Brotman and Moore (2008) found that eight grade female students identified fewer informal experiences with science than did male students, another limiting factor to why female students in particular may be less confident and in turn less interested in the physical science and engineering fields. Similar to Brotman and Moore, Archer et al. (2010) conducted focus groups with middle school students and also found girls had fewer experiences than boys and associated science with being dangerous. In addition, students from more affluent neighborhoods experienced more parent-guided informal science experiences compared to students in neighborhoods of lower SES.

Studies that address the perceptions of K-12 students regarding their perceptions of STEM careers and professionals suggest that students have little experience with these careers (Masnick et al. 2010). Inaccurate stereotypes of STEM professionals begin at an early age; one study found that elementary students commonly drew and described engineers as men, and someone who fixes things similar to a mechanic (Capobianco et al. 2011). The trend continues in the secondary years with students describing engineers as males who work on cars, fix or build things, work on trains, repair electronics, or generally design (Capobianco et al. 2011; Fralick et al. 2009). Similar to scientists and engineers, professionals in information technology (IT) careers are stereotyped as noncollaborative and monotonous by postsecondary students (Thomas and Allen 2006). Johnson and Miller (2002) assert that these careers are likely not attractive to students, particularly female students, because of the manner in which they are advertised (e.g., a men-only environment, long hours that would not be conducive to a family). They encourage that these careers in particular need to be more clearly articulated to school students to attract more diverse individuals.

Theoretical Framework

Lent et al. (1994, 2000) provide a promising theoretical framework and predictive model on interest and career choice, the social cognitive career theory (SCCT). The SCCT is based on Bandura's (1986) social cognitive theory of learning. According to Bandura, the most influential component to goal setting and action is self-efficacy, an individual's belief that he or she is capable of mastering events within his or her life. For example, a student with high science self-efficacy may believe that she is able to earn a good grade in her science class. Self-efficacy is involved in setting personal goals, analyzing decisions, and making commitments.

The theory also suggests that outcome expectations affect interest, when interacting with self-efficacy. For example, if a student feels that their success in mathematics will please their parents, they may study harder to achieve an A, and this success would encourage future interest in mathematics. Lent et al. (1994) connect Bandura's relationship between self-efficacy, outcome expectations and goals to contextual factors, personal inputs and interests to explain how individuals make career-related decisions (see Fig. 1). In the model, personal inputs are socially constructed factors, such as gender, background, race, and socioeconomic status; and intrapersonal factors, such as personality, that contribute to one's feelings of high or low self-efficacy. Contextual supports and barriers are external factors or individuals that either facilitate or impede high self-efficacy or setting academic or career goals. Studies that have used this framework encourage operationalizing these key aspects to be more relevant to the population that is being studied (Gushue 2006; Lent et al. 2008).
https://static-content.springer.com/image/art%3A10.1007%2Fs11165-013-9389-3/MediaObjects/11165_2013_9389_Fig1_HTML.gif
Fig. 1

The sociocognitive career theory (Lent et al. 1994, 2000)

The Current Study

Our goal was to develop the STEM-CIS based on key aspects of the SCCT (e.g., self-efficacy, outcome expectations, personal inputs, and contextual supports and barriers). Below, we discuss the stages of development of the STEM-CIS, as well as how this new instrument functioned in measuring factors related to middle school students' interest in and goals related to STEM subjects and potential careers.

Methodology

Clark and Watson (1995) identify six key components for developing scale items. For our study, we labeled each of their recommended steps as stages 1–6 to help clarify how we developed the STEM-CIS:
  1. Stage 1

    Conduct a literature review to help develop relevant items

     
  2. Stage 2

    Create a broad item pool of items that will test the target aspect

     
  3. Stage 3

    Preliminary pilot testing of items

     
  4. Stage 4

    Conduct structural analysis to determine which items are to be eliminated from the pool of items

     
  5. Stage 5

    Perform a factor analysis

     
  6. Stage 6

    Create subscales.

     

Context

The STEM-CIS was developed in the context of our STEM Career Awareness Project (NSF-funded ITEST project number 1031118) that emphasizes careers in the context of classroom instruction in rural, high-poverty districts (approximately 80 % free-and-reduced lunch) with approximately 85 % minority students (see Table 1 for demographic data). A survey instrument was needed to measure changes in student interest in STEM subjects and potential careers as a function of the intervention (Tyler-Wood et al. 2010).
Table 1

Demographics of participant schools

 

Free and reduced lunch

African American

White

Hispanic

American Indian

Other

School A

82.4

83.2

12.9

1.9

0.9

1.1

School B

85.1

71.6

24.6

2.5

0

1.3

School C

43.9

80.5

15.8

2.5

0.5

0.7

School D

76.6

67.7

16.7

3.4

9.2

6.0

School E

90.9

97.0

2.6

0

0

0.4

School F

59.4

40.5

51.4

2.8

0.4

4.9

School G

47.8

64.1

19.9

7.7

0

8.3

School H

51.9

42.6

38.6

7.4

1.7

9.7

www.schooldigger.com—school year 2010–2011. All numbers are in percentages

Participants in the Current Study

Data used in this validation process were collected from 1,061 students at seven middle schools in the rural southeastern USA (grades 5–8; see Table 1 for school demographics). For all students, it was the first time they had the seen the items. Students took the survey online using either an iPod touch® or a computer.

Instrument Development

Stage 1: Literature Review

The literature review consisted of a search for studies addressing students' interest in STEM and STEM careers, students' perceptions of STEM professionals, and social cognitive career theory applied in mathematics and science settings. The search included using ERIC, JSTOR, and Google Scholar and searching under the terms student STEM interest, instruments measuring STEM, students' perceptions of STEM, and social cognitive career theory and STEM, from the past 10 years. Individual disciplines within STEM were also searched to find literature regarding interest, perceptions of careers, measurements of interest, and social cognitive predictors of interest. Research from approximately 130 articles was accumulated to write a larger review of STEM education (Kier 2013), with only select references included in this manuscript. As previously described, findings suggested that interest in careers was often related to self-efficacy, outcome expectations, and previous learning experiences—addressed by Lent et al.'s social cognitive career theory. The literature review and theoretical framework guided the development of our initial pool of survey items, as well as other instruments measuring STEM courses and careers (e.g., Tyler-Wood et al. 2010). The literature indicated the need for a survey instrument that was developed for middle school students (Tyler-Wood et al. 2010; Usher 2009).

Stage 2: Create a Broad Item Pool of Items That Will Test the Target Aspect

Initial survey items were developed that connected to each of the aspects of the social cognitive career theory. We were guided by our literature review, a middle school mathematics and science self-efficacy scale (Fouad et al. 1997), and eight separate classroom STEM career discussions with sixth–eighth grade students at one of the schools participating in our development of the initial survey items with age-appropriate wording.

From this, we developed a 30-item instrument with 5 statements per SCCT aspect statements: questions about self-efficacy for mathematics, science, and technology were grouped together, as were questions related to student interest, outcome expectancy, and the other measured factors (see Table 2 for examples of original items). These statements were reviewed by three science educators, one faculty member in educational psychology, and a faculty member in counselor education with expertise in STEM career counseling. Based on these reviews, we decided to use a 5-point Likert-type scale [responses including “strongly disagree” (1), “disagree” (2), “neutral” (3) “agree” (4), and “strongly agree” (5)], so that the items were linked appropriately to all of the aspects of the social cognitive career theory and that the questions were understandable to middle school students.
Table 2

Examples from a previous version of the survey

Original questions numbers

SCCT aspects measured

Example question

Original Cronbach's alpha

1–5

Self-efficacy

I am able to get a good grade in science

0.762

6–10

Personal goals

I intend to enter a career that uses science

0.679

11–15

Outcome expectations

If I do well in my mathematics classes, it will help me in my future career

0.672

16–20

Interest

I am interested in careers that use science

0.613

21–25

Personal inputs

I would feel comfortable talking to people who work in mathematics careers

0.604

26–30

Contextual supports and barriers

I have people in my life that would help me get to a career in mathematics or science

0.579

Stage 3: Preliminary Pilot Testing of Items

These 30 survey items developed in stage 2 were piloted with 61 students in a diverse urban middle school that was not part of the final sample. Students completed the online survey using an iPod touch®, which took approximately 10 minutes.

Structural analyses from this pilot study found poor correlation scores between items and lower than anticipated alphas within each SCCT aspect. Several questions combined mathematics and science into a single item (e.g., It is too hard to get a career in mathematics and science; If I get good grades in mathematics and science, my parents will approve of me) and thus were eliminated or reworded due to the ambiguity as to which content area (mathematics or science) a student might have been responding.

Other problematic questions included some addressing student barriers in mathematics, science, and technology, which had very low item–total correlations (ranging from 0.05 to 0.16), and were thus eliminated. At this point, for the sake of conceptual clarity of individual scales, we restructured our survey into parallel sets of items addressing each subject area of science, mathematics, and technology, and added another question about being supported in these subject areas. This led to the organization of 11 questions that addressed six social cognitive career factors: self-efficacy (2), outcome expectations (2), goals (2), interests (2), contextual supports (2), and personal disposition (1) conducive to finding out more information about a career in the field. These restructured subscales and items were piloted again with three middle schools. The demographics of these schools are summarized in Table 3.
Table 3

Demographics of pilot schools

 

Free and reduced lunch

African American

White

Hispanic

American Indian

Other

School A

82.4

83.2

12.9

1.9

0.9

1.1

School F

59.4

40.5

51.4

2.8

0.4

4.9

School H

51.9

42.6

38.6

7.4

1.7

9.7

www.schooldigger.com—school year 2010–2011. All numbers are in percentages

Stage 4: Conducting Structural Analysis to Determine Which Items Are to be Eliminated from the Pool of Items

We were satisfied with the internal consistency estimates of the subscales (described below), but an initial confirmatory factor analysis led to a modification of two questions in the technology subscale. We piloted these technology questions with a newly developed engineering scale, worded consistently with the other three subscales. These questions were piloted with 102 students from schools A, B, D, E, and G. The engineering scale alone was also piloted at school H, with 148 students. This totaled 250 students who took the engineering scale and 102 students who responded to the technology scale with the two revised items. As before, online surveys were completed by students using an iPod Touch, with average administration time being approximately 10 minutes. The many stages of piloting and revising the subscales explains the difference in sample sizes when doing the final measures of internal consistency and confirmatory factor analyses (Tables 4 and 5).
Table 4

Item correlations for STEM-CIS

Item number

Alpha or item–total correlation

Social cognitive career theory aspect

Itema

Science α = 0.77, average scale score = 27.61, SD = 7.19, N = 831

 S1

0.42

Self-efficacy

I am able to get a good grade in my science class

 S2

0.47

Self-efficacy

I am able to complete my science homework

 S3

0.48

Personal goal

I plan to use science in my future career

 S4

0.48

Personal goal

I will work hard in my science classes

 S5

0.48

Outcome expectation

If I do well in science classes, it will help me in my future career

 S6

0.27

Outcome expectation

My parents would like it if I choose a science career

 S7

0.43

Interest in science

I am interested in careers that use science

 S8

0.49

Interest in science

I like my science class

 S9

0.30

Contextual support

I have a role model in a science career

 S10

0.44

Personal input

I would feel comfortable talking to people who work in science careers

 S11

0.29

Contextual support

I know of someone in my family who uses science in their career

Mathematics α = 0.85, average scale score = 25.17, SD = 8.52, N = 829

 M1

0.57

Self-efficacy

I am able to get a good grade in my mathematics class

 M2

0.59

Self-efficacy

I am able to complete my mathematics homework

 M3

0.60

Personal goal

I plan to use mathematics in my future career

 M4

0.57

Personal goal

I will work hard in my mathematics classes

 M5

0.62

Outcome expectation

If I do well in mathematics classes, it will help me in my future career

 M6

0.43

Outcome expectation

My parents would like it if I choose a mathematics career

 M7

0.52

Interest in mathematics

I am interested in careers that use mathematics

 M8

0.57

Interest in mathematics

I like my mathematics class

 M9

0.36

Contextual support

I have a role model in a mathematics career

 M10

0.55

Personal input

I would feel comfortable talking to people who work in mathematics careers

 M11

0.44

Contextual support

I know someone in my family who uses mathematics in their career

Technology α = 0.89, average scale score = 26.27, SD = 10.41, N = 102

 T1

0.60

Self-efficacy

I am able to do well in activities that involve technology

 T2

0.38

Self-efficacy

I am able to learn new technologies

 T3

0.71

Personal goal

I plan to use technology in my future career

 T4

0.68

Personal goal

I will learn about new technologies that will help me with school

 T5

0.63

Outcome expectation

If I learn a lot about technology, I will be able to do lots of different types of careers

 T6

0.71

Outcome expectation

When I use technology in school, I am able to get better grades

 T7

0.68

Interest in technology

I like to use technology for class work

 T8

0.57

Interest in technology

I am interested in careers that use technology

 T9

0.59

Contextual support

I have a role model who uses technology in their career

 T10

0.55

Personal input

I would feel comfortable talking to people who work in technology careers

 T11

0.53

Contextual support

I know of someone in my family who uses technology in their career

Engineering: α = 0.86, average scale score = 30.03, SD = 8.85 , N = 250

 E1

0.59

Self-efficacy

I am able to do well in activities that involve engineering

 E2

0.60

Self-efficacy

I am able to complete activities that involve engineering

 E3

0.61

Personal goal

I plan to use engineering in my future career

 E4

0.54

Personal goal

I will work hard on activities at school that involve engineering

 E5

0.59

Outcome expectation

If I learn a lot about engineering, I will be able to do lots of different types of careers

 E6

0.53

Outcome expectation

My parents would like it if I choose an engineering career

 E7

0.66

Interest in engineering

I am interested in careers that involve engineering

 E8

0.62

Interest in engineering

I like activities that involve engineering

 E9

0.45

Contextual supports

I have a role model in an engineering career

 E10

0.59

Personal input

I would feel comfortable talking to people who are engineers

 E11

0.36

Contextual support

I know of someone in my family who is an engineer

aItem choices were on a Likert-type scale, 1 (strongly disagree), 2 (disagree), 3 (neither agree nor disagree), 4 (agree) and 5 (strongly agree)

Table 5

Summary of confirmatory factor analyses

 

Parameters estimated

df

Chi square

CMIN/df

NFI

CFI

RMSEA

Science only (N = 1061)

49

28

59.45

2.12

0.97

0.98

0.033

Mathematics only (N = 1061)

49

28

55.78

1.99

0.98

0.99

0.031

Technology only (N = 1056)

42

35

70.46

2.01

0.97

0.99

0.031

Engineering only (N = 1061)

39

38

50.27

1.32

0.95

0.99

0.017

All four factors (N = 1061)

215

819

1,745.92

2.13

0.84

0.91

0.033

All items on one factor (N = 1061)

285

749

1,356.70

1.811

0.88

0.94

0.028

In all models, means and intercepts were estimated in order to allow AMOS to handle missing data appropriately. This led to higher numbers of parameters estimated than other models but utilized the entire N = 1061 sample for each analysis. The exception is in the Technology subscale, where five cases with Mahalanobis D2 were removed from the analysis

Stages 5 and 6: Confirmatory Factor Analysis of Subscales

To verify the psychometric properties of each subscale (i.e., whether each subscale was one-dimensional with desirable model fit), each content scale (science, mathematics, technology, and engineering) was subjected to confirmatory factor analysis via AMOS.1 For each subscale analysis, a single-factor solution was modeled using data from cases with complete data on those subscale items. Modification indices were examined to optimize the fit of the initial model.2 This resulted in slightly different parameters estimated for each content scale. Once a model was specified appropriately, the full data were imported into the model. AMOS can utilize cases with partial information (i.e., it does not require complete data for all cases and, as such, does not delete cases with partial missing data from the analysis) and all CFA analyses reported below are therefore based on the full sample of N = 1,061.

Science Subscale

For the science subscale analyses, Mahalanobis D2 was used to evaluate multivariate outliers.3 In this scale, a reasonable cutoff for D2 with 38 parameters estimated would have been 53.38. No outliers with a D2 over 36 were observed. Testing model fit in confirmatory factor analyses (CFAs) routinely involves evaluating and reporting multiple fit indices, as each one captures different aspects of model fit. The chi square statistic is the first measure of how well a model fits the data; however, it is greatly influenced by sample size and thus is rarely found to be nonsignificant in samples sufficiently large enough to legitimately perform CFA within (Thompson and Daniel 1996).

Therefore, an appropriate model fit was assessed using the root mean square error (RMSEA), the comparative fit index (CFI), and the normed fit index (NFI). Model fit is usually considered adequate when the following criteria are met: RMSEA should be <0.08, NFI should meet or exceed 0.90, and CFI should meet or exceed 0.954 (Byrne 2010; Hu and Bentler 1999; Marsh et al. 2004). Model chi squared generally only is used for comparing nested models, as it is determined both by sample size and goodness of fit, and CMIN/df is a commonly reported scaling of chi square, divided by the degrees of freedom in the model. In general, for both of these indices, smaller numbers are better, but there is no widely accepted way to interpret them objectively.

In Table 5 and Fig. 2, we summarize this model. As you can see in the table, the model fit was strong, indicating that this subscale indeed represents a single coherent factor. In Fig. 2, you can see that standardized factor loadings ranged from 0.22 to 0.65. Correlations between error terms represent correlations between scale items and are not generally interpreted substantively.
https://static-content.springer.com/image/art%3A10.1007%2Fs11165-013-9389-3/MediaObjects/11165_2013_9389_Fig2_HTML.gif
Fig. 2

Confirmatory factor analysis for science

Mathematics Subscale

Analysis of the mathematics subscale was equally positive. Mahalanobis D2 was examined to evaluate multivariate outliers, and again, no multivariate outliers were observed. The results of this analysis are presented in Table 5 and Fig. 3.
https://static-content.springer.com/image/art%3A10.1007%2Fs11165-013-9389-3/MediaObjects/11165_2013_9389_Fig3_HTML.gif
Fig. 3

Confirmatory factor analysis for mathematics

As you can see in Table 5, model fit for this scale was strong, exceeding all recommendations for NFI, CFI, and RMSEA, indicating that this subscale indeed represents a single coherent factor. Further, as Fig. 3 shows, standardized factor loadings were generally strong, ranging from 0.29 to 0.70.

Technology Subscale

Examination of Mahalanobis D2 statistics for the technology subscale indicated five cases in excess of 60, which subsequently were removed as excessive multivariate outliers, leaving us N = 1,056 individuals retained for analysis. The results of this analysis are presented in Table 5 and Fig. 4. As you can see in Table 5, this model fits the data well, with fit indices indicating excellent fit. Standardized factor loadings, presented in Fig. 4, ranged from 0.51 to 0.95, again indicating that this subscale represents a single coherent factor.
https://static-content.springer.com/image/art%3A10.1007%2Fs11165-013-9389-3/MediaObjects/11165_2013_9389_Fig4_HTML.gif
Fig. 4

Confirmatory factor analysis for technology

Engineering Scale

Examination of Mahalanobis D2 indicated that no outliers were observed, and thus all individuals were retained for analysis of the engineering subscale. This analysis is summarized in Table 5 and Fig. 5. Model fit for this scale was strong despite the relatively small sample size (N = 282) and standardized factor loadings ranged from 0.33 to 0.78, indicating that this subscale represents a single coherent factor.
https://static-content.springer.com/image/art%3A10.1007%2Fs11165-013-9389-3/MediaObjects/11165_2013_9389_Fig5_HTML.gif
Fig. 5

Confirmatory factor analysis for engineering

Discussion

The STEM Career Interest Survey developed in this study was shown to be psychometrically sound for each of the subscales of science, technology, engineering, and mathematics. Because some researchers will be interested in student interest toward more than one of these content areas, we also explored how these four subscales worked together as a single measure. This is a reasonable approach as analyses showed the four latent subscale scores to be strongly correlated; correlations between latent factors ranged from r = .72 to .82. Thus, we estimated two other models: all four subscales in a single model as four separate but correlated factors and all items from all four subscales characterized as one single factor (Appendix).

Because of the complex sampling history of this project, large amounts of imputation would be undesirable. Thus, these models were estimated using all 1,061 students but without imputing missing data. This has the desirable effect of not relying upon hundreds of imputed engineering scale scores, for example, but has the undesirable side effect of not allowing the model to be fit using modification indices, and thus, the model fit for these last two models is probably lower than otherwise might be achieved. We also were not able to examine the data for outliers, again, which could harm model fit substantially.

Table 5 shows reasonable model fit for the four-factor model. In particular, RMSEA indicates strong model fit, despite NFI and CFI falling slightly below norms for strong models [recall Marsh et al. (2004) argues that strict adherence to cutoff rules is not defensible]. Thus, it is our conclusion that it is defensible to use this instrument in middle grade populations with all four subscales characterized by four separate subscores.

For researchers who seek one single overall score from these four subscales (or from several of them), the evidence from the single-factor model that included all items from all four scales loading on a single factor shows reasonable model fit, with a RMSEA well below norms and CFI and NFI that are reasonably close to established guideposts for acceptable model fit. This model fit is also significantly better than the four-subscale model (ΔΧ2 = 389.22, df = 70, p < .000000000001; ΔCFI = 0.03, ΔNFI = 0.04, and ΔRMSEA = 0.005). This final analysis supports combining all items into a single overarching scale score, should researchers find that desirable (see Table 5).

Thus, it is possible to use this scale in a variety of ways, each defensible and supported by these analyses. It is defensible to use any of the four subscales individually, to use more than one (or even all four) together in a single instrument, calculating individual subscale scores for each content area, and it is also defensible to use more than one subscale and combine all items into a single overall score reflecting career interests in the STEM field overall.

Limitations of the Current Study

There are myriad limitations to any study as ambitious as this one. The first and most obvious limitation involves the sample. Any psychometric property of a scale is contingent on the sample—in other words, your results might differ from ours using the same scale and a different sample. In future work, we expect to evaluate the stability of the model using cross-validation and invariance analyses.

Another limitation is that we have not reported any analyses that speak to whether this scale is equally appropriate for both girls and boys (or for different racial/ethnic or age groups). It is reasonable to wonder, for example, whether girls and boys have equally strong internal consistency coefficients, or identical factor structures. These types of analyses, invariance analyses, are lengthy and complex to report, and thus, for the sake of brevity, we have chosen not to report them in this paper. They also ideally require much larger and more diverse samples, which we are currently engaged in collecting. Initial analyses (not reported here) indicate strong invariance across gender groups. We anticipate reporting these results more fully in a subsequent paper.

Finally, as has been noted above, our sampling framework was somewhat complex and driven by the processes involved in the execution of the grant that funded the data collection and the realities of data collection with real students in high-poverty school districts.

Conclusions

The STEM-CIS was developed to measure the effects of strategies intended to promote the awareness of, interest in, and intent to pursue STEM careers with rural, minority, middle school students (Blanchard et al. 2012). Its development thus addresses calls for an age appropriate measure (Gushue 2006; Lent et al. 2008) that is theoretically strong (Whitfield et al. 2008). With this particular population, we were interested in finding out not only if students were interested in STEM subjects and careers but also what factors influenced this interest.

The development of this survey built on previous instruments (Fouad et al. 1997; Tyler-Wood et al. 2010), as well as a promising framework, the social cognitive career theory (Lent et al. 1994, 2000). The SCCT has been used and psychometrically evaluated in predicting interest with middle school students and now has been applied to this new STEM career interest survey.

Implications

The STEM Career Interest Survey developed in this study was shown to be psychometrically sound and able to be used by researchers or professional developers in science, technology, engineering, and mathematics, using one or more subscales or all of them as one instrument, as needed. As such, we expect it will be beneficial to researchers, professional developers, and evaluators in measuring STEM career interest and the effects of STEM programs on changes in student interest in STEM subjects and careers. The knowledge that we gain from the use of this instrument may help to inform efforts taken at the middle school level as we seek to increase students' interest in STEM subjects, majors, and careers. We also plan to validate this instrument with other populations of students in urban settings and at the high school level. Innovations worldwide depend on highly qualified professionals in STEM careers (Hill et al. 2010; Regisford 2012; White House Office of Science and Technology Policy 2012). We hope that the STEM-CIS will make a contribution to studying the impact of our efforts to attract students to meet these needs.

Footnotes
1

AMOS is the structural equation modeling software produced with IBM/SPSS.

 
2

Modification indices are reported in AMOS as chi square values with one degree of freedom. As such, a significant improvement in the model is signified by a chi square in excess of 3.84. In order to be very conservative with this process, and to model inter-item correlations that contributed substantially to model fit, we used a rule that we would examine inter-item correlations that resulted in a modification index (MI) of 8.00 or greater. Further, note that because these models were all single latent variable models with multiple measured variables that legitimately should be allowed to correlate, there is no conceptual issue with allowing items designed to measure a particular latent variable to correlate. We cannot model all possible correlations between items on a scale in CFA due to lack of degrees of freedom, and thus, the use of modification indices is merely an efficient method of identifying those items that, by allowing a nonzero correlation, significantly and substantially (and appropriately) improve model fit.

 
3

Mahalanobis D2 is a common index of whether an individual score is aberrant within the multivariate distribution of scores; higher numbers indicate the score is farther from the center of the multivariate distribution. Guidelines for assessing Mahalanobis D2 suggest that a reasonable cutoff score is a chi square value that would be significant at p < .05 for the number of df equal to the number of variables or parameters estimated in the model.

 
4

Note that Marsh et al. (2004) argue against strict cutoff scores or “golden rules,” and thus, these indices should be treated as the continuous variables they are, and interpreted in context.

 

Acknowledgments

The authors wish to thank Michael D. Cobb for his helpful suggestions on the initial development of this instrument and all of the collaborators on this project who participated in this research. This research was funded by an ITEST grant (2010–2013) from the National Science Foundation (award number 1031118). The opinions expressed are those of the authors and do not represent the views of the National Science Foundation or North Carolina State University.

Copyright information

© Springer Science+Business Media Dordrecht 2013