Keywords

1 Introduction

Policymakers and scientific leaders in the United States (US) have prioritized the cultivation of a diverse science, technology, engineering, and mathematics (STEM) workforce in the US [1–3]. In its 2010 report Preparing the Next Generation of STEM Innovators, the National Science Board offered two mutually reinforcing observations. First, the US’s long-term prosperity is dependent upon “talented and motivated individuals who will comprise the vanguard of scientific and technological innovation.” Second, every student in the US “deserves the opportunity to achieve his or her full potential” [4]. In short, excellence and equity in STEM education are interrelated.

This goal can be realized only if underrepresented groups receive a larger proportion of the nation’s STEM degrees. Educators and policymakers long have empha-sized the need to overcome disparities of race, ethnicity, gender, and socioeconomic status in realizing equality and diversity in STEM fields, and rightfully so. However, Americans with disabilities historically have been excluded from postsecondary STEM education and remain underrepresented in the STEM workforce.

2 Broadening Participation in STEM

The National Science Foundation’s (NSF) Research in Disabilities Education (RDE) program has sponsored research and development projects to broaden participation and increase achievement of people with disabilities in STEM education and the STEM workforce. At the heart of these efforts has been RDE’s “Alliances for Stu-dents with Disabilities in STEM” (Alliances) project track. Intended to serve a specific geographic region, Alliances involve multiple institutions of higher education and secondary school systems working as a team “to employ evidenced-based practices and promising interventions to advance students across critical academic junctures, to degree completion, and into the workforce or graduate STEM degree programs.” Taken together, Alliances create a unified program of change extending beyond academia to include industry and government research experiences for students with disabilities. In addition, Alliances typically go beyond matters of STEM content knowledge to focus on underlying issues affecting the differential learning, participation, retention and graduation rates of postsecondary students with disabilities in STEM.

NSF has funded 10 RDE Alliances to across the US, each of which is tasked with serving its specific region or state. The primary mission of the Alliances is to support the advancement of students with disabilities from high school into college (including transfer from two-year to four-year colleges), from undergraduate STEM degree programs into graduate STEM education, and from postsecondary education into the STEM workforce. Emphasis is placed upon successful navigation of the critical transition points between secondary education, undergraduate education, graduate education, and employment [5]. Stated Alliance goals typically focus on recruitment to, retention within, and progression through project activities, as well as graduation of students with disabilities from secondary and two-year and four-year postsecondary education.

At the same time, Alliances are also charged with establishing scalable and replicable models to demonstrate how comprehensive, multidisciplinary networks of high schools, 2- and 4-year undergraduate institutions, and graduate programs may broaden participation of students with disabilities within STEM education and the STEM workforce. Alliances also may pilot novel interventions to serve as promising practices.

3 Establishing a Theory of Change

Researchers concerned about the evaluation of higher education have argued that the basic availability of data should not dictate the approach for undertaking such evaluations [6]. If evaluation is to determine educational efficacy, then it must be considered from the beginning through the development of meaningful indicators and the provision of data collection and analysis for such ends. There exists a voluminous and robust scholarship around the selection of indicators for the evaluation of education, especially at the more complex, system level [7–11] (Fig. 1).

Fig. 1.
figure 1

Overall theory of change for BreakThru

Full size image

To determine the efficacy of BreakThru, lead investigators have worked closely with project evaluators to identify a theory of change underlying the project, including its interventions and activities and progress toward stated goals. Based on the framework for theory-driven evaluations [12], the evaluation questions have driven the development of BreakThru’s Theory of Change, which is informed primarily by the project’s emphasis on e-mentoring.

Most broadly, BreakThru holds that successful recruitment into the project will lead to participation in e-mentoring activities at the heart of BreakThru as well as broader increases in STEM learning and support received by students with disabilities. Participation in these activities will result in positive changes in student’s internal characteristics and enable them to persist through STEM education, thereby accomplishing BreakThru’s intended effect of increased graduation among students with disabilities. (Fig. 2).

Fig. 2.
figure 2

Expanded theory of change for BreakThru, detailing the measures and key constructs for determining efficacy of project activities and progress toward objectives. Color-coded tabs outline instruments and data sources for each measure.

Full size image

A more detailed examination of the theory of change demonstrates specific activities expected to contribute, indirectly and directly, to increased persistence among students with disabilities within STEM education. Equally important are survey instruments, as well as constructs derived from instrument items, and other data sources, such as enrollment and graduation records, for establishing evidence of persistence.

3.1 Recruitment and E-Mentoring Activities

E-mentoring activities comprise the heart of BreakThru are supported through targeted recruiting efforts and STEM learning and support activities. Toward this end, project evaluation and its theory of change are informed by the manner in which students are recruited and their reasons for joining the project.

The purpose of BreakThru’s e-mentoring is to foster a relationship through which experienced persons share knowledge and perspective, as mentors work to achieve the personal and educational growth of students through digital communication, including a variety of social media and online platforms to support mentoring relationships. This type of mentoring is known as electronic mentoring or, simply, “e-mentoring” to distinguish it from traditional face-to-face mentoring. As e-mentoring can occur asynchronously and remotely, it offers benefits not associated with conventional mentoring. A growing literature has described the usefulness of e-mentoring in educational, business, human resources, and social environments. Unfortunately, while conceptual models have been developed to describe the e-mentoring relationship, there is limited empirical evidence for their overall effectiveness. For students with disabilities interested in STEM careers, the use of e-mentoring allows for more access to mentors representing a variety of disciplines and locations. Project evaluation takes into account and measures, to the extent possible, dosage (i.e. frequency and duration) of e-mentoring activities, communication media used within the project (e.g., SecondLife, e-mail, telephone), as well as key evaluation measures of communication satisfaction, relationship satisfaction, and participants’ perceptions of student personal responsibility.

In addition to e-mentoring activities, the theory of change also takes into account STEM learning strategies such as time management, study strategies, reading skills, and dealing with text anxiety through completion of 25 student learning modules created as part of the project and frequently completed by project mentors and mentees together.

3.2 Changes in Internal Characteristics

Participation in e-mentoring activities and complementary STEM learning and sup-port activities such as the learning modules are expected to result in positive changes as mentees internalize the support they have received from e-mentoring relationship and knowledge received through module completion, among other project activities. Changes in internal characteristics may be assessed by six measures:

Self-advocacy. – In response to concerns that overprotection of people with disabilities by authority figures and professionals resulted in dependency and undermined autonomy, self-advocacy emerged as a key concept within disability and educational research for the successful transition of students with disabilities toward independent adulthood [13–15]. Toward this end, Test et al. [16] offered a conceptual framework specifically for students with disabilities. In their conception, self-advocacy is composed of four overarching components: knowledge of self, knowledge of rights, communication, and leadership, each of which is guiding by a series of subcomponents. In addition, the measurement of self-advocacy among students with disabilities has been informed by instruments such as the General Self-Efficacy (GSE) Scale [17, 18], which query individuals’ perceptions about their ability to solve problems, achieve goals, cope with difficulties, and overcome challenges.

BreakThru accepts self-advocacy as “the ability to effectively communicate, con-vey, negotiate or assert one’s interests, desires, needs, and rights” [19]. Self-advocacy is measured by a 12-item scale within the Student Survey, drawn primarily from the Self Determination Student Advocacy Skills (SDSAS) Questionnaire [20].

Self-determination. – The ability of people with disabilities to make decisions and attain goals underlie the principles of self-determination. Self-determination is closely related to self-advocacy, but whereas self-advocacy stresses speaking and acting on one’s behalf, self-determination emphasizes actual decision-making processes and control of one’s life. Algozzine et al. [21] defined self-determination as “the combination of skills, knowledge, and beliefs that enable a person to engage in goal-directed, self-regulated, autonomous behavior” (219) and identified the requisite skills associated with effective self-determination, such as decision making, goal setting, and problem solving. This initial work was elaborated by Cobb et al. [22] and others, and a number of scales have been offered to measure self-determination [20, 23].

BreakThru accepts self-determination as “the ability to act as the primary causal agent in one’s life and set goals and make decisions that are unrestricted from undue external influence or interference.” This also involves making informed decisions and taking responsibility for those decisions [15]. Self-determination is measured by a 17-item scale within the Student Survey drawn from the AIR Self Determination Scale [24] and the Self-Determination Student Scale [25].

Science and Math Anxiety. – Science affect may be defined as perceptions or emotions associated with science. Previous research has found that science affect is significantly correlated with reported high school preparation in science and college science GPA [26]. There are two measurable aspects of science affect: Negative science affect involves anxiety about performing well on science exams and uneasiness when doing a science experiment. Positive science affect involves perceiving science as interesting and enjoyable and useful for one’s career. Science affect is measured by a 16-item scale within the Student Survey.

Math affect similarly may be understood as perceptions or attitudes associated with mathematics. Math affect has been shown to involve complex factors such as feelings of pressure, performance inadequacy and test anxiety that interfere with the ability to solve math problems [27]. There are two measurable aspects of math affect: Negative math affect involves anxiety about solving problems, general confusion, and uneasiness when solving problems. Positive math affect entails perceiving math as interesting and useful for one’s future. Math affect is measured by a 14-item scale in the Student Survey.

Intention to Persist. - the likelihood to persist by pursuing more education or a career in STEM. This 8-item scale is adapted from Toker’s (2010) scale that measures short-term commitments and long-term commitments as they apply to education, as well as degree attainment intentions. The scale is also informed by Williams, Wiebe, Yang, Ferzli, & Miller (2002).

3.3 Intended Program Effect

Increased intention to persist is expected to result in concrete steps to persist, such as applying to and enrolling in graduate school, participating in a STEM-related internship, or obtaining a STEM-related job. Open-ended responses are collected from project mentors and mentees as part of monthly reporting and qualitatively analyzed for evidence of concrete steps to persist. In addition, actual persistence is measured by educational attainment and successful navigation of critical transition points for STEM careers.

4 Results

The Student Survey to assess changes in students’ internal characteristics and intention to persist has been administered over the lifespan of the BreakThru since Fall 2011 to present. The following tables summarize changes from “pre,” when the student first completed the survey upon induction into the project, to “post,” when the student last completed the survey. As of Spring 2014, 29 postsecondary participants had completed the student surveys for “pre” and “post” analysis. (Table  1).

Table 1. Internal characteristics constructs for postsecondary students enrolled in BreakThru between Fall 2011 and Fall 2014, with change over time and statistical significance.
Full size table

The construct scores are presented using 5-point Likert scale (1, Strongly Disagree to 5, Strongly Agree) based upon the Student Survey items. The higher the score, the greater the indication of progress for that construct. For assessment purposes, negatively worded construct items were reverse-coded.

5 Discussion

Over the duration of the project, secondary and postsecondary students participating in BreakThru have experienced improvements in internal characteristics related to persistence in STEM education, as measured by the project’s evaluation instruments. Among postsecondary students, all constructs except intention to persist were statistically significant. Of these, gains in self-advocacy were highest, suggesting that e-mentoring activities and targeted modules on improving self-advocacy and putting it into practice have resulted in positive changes in students’ internal characteristics. While not as great of a change, self-determination similarly increased over the duration of the project. A more nuanced examination of responses to individual items reveals that students feel more capable of determining their own strengths and weaknesses, even if do not necessarily feel comfort with the material overall. They show the largest increase when evaluating their ability to check over their work.

Regarding science and math anxiety, results have been somewhat more mixed. Decreases in negative affect have been offset somewhat by decreases in positive affect, making it necessary to examine individual items for a more nuanced interpretation of results. For scale items related to science affect, for example, increases in positive affect were seen on many items, with the notable exceptions of students’ enjoyment in learning and their desire to do better than classmates. Of particular concern for negative affect are student anxiety levels and concern about failing tests. Regarding math affect, improvements were seen in a number of areas, including students’ interest in taking more classes in the future and their enjoyment in learning mathematics.

Despite these somewhat mixed findings regarding affect, longitudinal findings from the Student Survey that informs the Theory of Change suggests the efficacy for e-mentoring in improving persistence in STEM education. It also is worth noting that both “pre” and “post” scores are above average in terms of the Likert-type scale being utilized. Most constructs were close to a score of 4.00, which constitute general agreement with construct items.

6 Conclusions

E-mentoring activities at the heart of BreakThru may represent a promising approach for improving student persistence through STEM education and for navigating critical transition points to STEM employment. The purpose of the theory of change presented in this paper is to establish a promising means for documenting the efficacy of these activities and their effect on student intention to persist and concrete steps to persist. We hope to inform the field at large about ways to assess and evaluate the efficacy of these technologically mediated activities.

Acknowledgements. BreakThru is a project of the Georgia STEM Accessibility Alliance, which is supported by the National Science Foundation (NSF), Research in Disabilities Education (RDE) Awards # 1027635 and 1027655. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

The authors wish to acknowledge the contribution of Dr. Tom McKlin and Dr. Shelly Engelman of SageFox Consulting Group for their external evaluation services. The authors also acknowledge Ms. Jana Thompson and Mr. Scott Pollack at the University of Georgia and Dr. Meltem Alemdar at the Georgia Institute of Technology for their internal evaluation support.