Since the 1980s, US educators and governmental agencies have attempted to broaden participation in science, technology, engineering, and mathematics (STEM) fields by creating pipelines to increase the number of underrepresentedFootnote 1 groups entering the workforce (James & Singer, 2016; Malcom & Feder, 2016). Yet women and racially minoritized students continue to be underrepresented in certain STEM disciplines (Riegle-Crumb et al., 2012; Sax et al., 2017) and graduate at inequitable rates compared to white students (Riegle-Crumb et al., 2019).

One promising avenue to broaden participation and enhance equitable outcomes in STEM is to use high-impact practices (HIPs). Kuh’s (2008) seminal report combined transformative practices together under a common name based on their association with positive student learning outcomes and retention (Kilgo et al., 2015; Schwartz & Miller, 2020). Subsequently, scholars have focused on the positive outcomes associated with specific HIPs (e.g., Andrade, 2008; Binder et al., 2015), and for specific groups of underrepresented students (e.g., Finley & McNair, 2013; Kinzie et al., 2021; Swaner & Brownell, 2009).

While there is little research on HIPs, in combination, in STEM undergraduate education, scholars have found positive relationships between specific HIPs, such as undergraduate research, and student outcomes (e.g., Hernandez et al., 2018; Hunter et al., 2006; Linn et al., 2015). Undergraduate research has also been shown to foster more equitable access (Bangera & Brownell, 2014) and thus broaden participation in STEM (Hernandez et al., 2018). Studies on other HIPs, including community-based learning, internships, and capstones, have found participation is positively related to STEM recruitment, retention, and graduation (e.g., Dinh & Zhang, 2021; Meador, 2018). Yet racially minoritized students still participate in HIPs at lower rates than their white counterparts in STEM (e.g., Minichiello et al., 2021), indicating systematic barriers in access.

Thus, while HIPs can broaden participation in STEM, they can also fall short without attention to equity. Scholars have recently started to call for a critical examination of whether and how HIPs advance educational equity for racially/ethnically minoritized students (Greenman et al., 2022; Kinzie et al., 2020; Kinzie et al., 2021; Peters et al., 2019). Their claim is that it is important for institutions to use an equity-minded lens in examining the ways in which HIPs can dismantle barriers to broadening participation in STEM, rather than viewing HIPs as a simple checklist (Finley, 2019; Harper, 2010). Equity-minded work aims to refocus efforts from trying to change student inputs and behaviors, to changing policies, practices, and beliefs of faculty and staff in higher education (Bensimon, 2007). Some scholars have offered alternative HIP models to better serve our most vulnerable students (Stewart & Nicolazzo, 2018) and HIP programs that center equity as a linchpin (Zilvinskis et al., 2022). However, we know little about how federally funded programs purportedly aiming to broaden participation can serve as a mechanism to advance equity.

The purpose of this study is to bring an equity-minded lens to an examination of HIPs that aim to broaden participation in undergraduate US STEM education, including their prominence, design, implementation, and outcomes. We developed a framework to examine National Science Foundation (NSF)-funded programs, as broadening participation is an increasing investment area for the NSF (James & Singer, 2016) yet we know grantmaking often reproduces the racial status quo (McCambly & Colyvas, 2022). We conducted a systematic review of 38 NSF-funded programs that included HIPs, drawing on multimodal data from the STEM for All Multiplex and survey data, to answer research questions on what HIPs were prominent, how the HIPs were designed to ensure equity-mindedness, if at all, what characteristics were used in implementation of the HIPs, and what outcomes were measured to determine HIP impact. Federal funding is often a key mechanism by which institutions can develop and implement HIPs, yet it is an underexplored area of the literature. Therefore, the findings of this study contribute to the field by examining the equity-mindedness of NSF-funded programs to broaden participation in STEM that include HIPs. We highlight two exemplar programs and offer recommendations for researchers and funders to more effectively implement equity-minded HIPs to broaden participation in undergraduate STEM education.

Conceptual framework

We grounded this study in an HIP framework (Fig. 1) derived from Kuh’s (2008) 11 HIPs, and two related indices of equity-minded HIP design (Greenman et al., 2022; Zilvinskis, 2019) and implementation (Kinzie et al., 2021; Kuh et al., 2017; Kuh & O'Donnell, 2013) that influence HIP outcomes and impact.

Fig. 1
figure 1

Conceptual framework overview

Full size image

High-impact practices

Kuh (2008) conceptualized HIPs through examining and synthesizing the student involvement and effort literature on collegiate practices, programs, and services. What stood out about these curricular, co-curricular, and pedagogical experiences were the positive benefits to students when they participated in one or more HIP, such as higher levels of engagement and personal and social development (Finley & McNair, 2013; Kuh, 2008). Over time, scholars have identified 11 HIPs (refer to Table 1).

Table 1 High-impact practices
Full size table

Equity-minded design characteristics of HIPs

Scholars have found racial disparities in HIP access and participation (Finley & McNair, 2013; Greenman et al., 2022; Kinzie et al., 2020) and inequitable outcomes in specific HIPs (Zilvinskis, 2019). For example, in a survey of 576 engineering and computer science undergraduates at two land-grant institutions, 43% of students reported not engaging in any HIP, while Latinx (65%) and other non-White (61%) students reported the highest levels of non-participation compared to other groups (Minichiello et al., 2021). Similarly, in a national survey, Zilvinskis (2019) found Black students had qualitatively different experiences in HIPs compared to other students. For example, when participating in undergraduate research, they reported lower levels of a supportive environment compared with other racial and ethnic groups.

Scholars have pointed to the inequity in HIPs as an outcome of the whiteness that undergirds these practices (Kinzie et al., 2021; Patton et al., 2015; Stewart & Nicolazzo, 2018). As a field, we know little about how HIPs serve underrepresented students (Swaner & Brownell, 2009). Scholars have called for HIPs to be more equity-minded in design (Greenman et al., 2022), meaning that program designers should critically examine the knowledge, beliefs, assumptions, and epistemologies undergirding the HIPs. Thus, designing equity-minded HIPs is not only about reducing inequity in participation rates, but re-framing HIPs to be more culturally relevant (Zilvinskis et al., 2022), considering the perspectives of underrepresented students rather than trying to serve the “traditional” student body (Stewart & Nicolazzo, 2018). For the equity-minded design index of our framework, we drew on Greenman et al.’s (2022) synthesis of proposed solutions to make HIPs more equitable and Zilvinskis’ (2019) study on underrepresented students in HIPs, resulting in six characteristics (Table 2).

Table 2 Equity-minded design characteristics of HIPs
Full size table

Implementation characteristics of HIPs

While HIPs have proven to be effective, the quality and thus the impact of these practices depend on how they are implemented (Finley & McNair, 2013; Greenman et al., 2022; Kinzie et al., 2020; Kuh et al., 2017; Zilvinskis, 2019). Kinzie et al. (2020) found that students experienced differing levels of quality across HIPs. “Most of the research about HIPs does not take into account the structural aspects of the program or practice or how well specific high-impact practices are implemented” (Kuh & O'Donnell, 2013, pp. 1-2). For the implementation index of our framework, we drew on the literature to develop 11 characteristics (Table 3).

Table 3 Implementation characteristics of HIPs
Full size table

Outcome measures

Most HIP research draws on conventional measures for determining impact, such as skills learned and how the program adds value to the students’ learning and development (Caspersen et al., 2017; Kilgo et al., 2015). However, inequitable environmental contexts can undermine the positive impacts of HIPs on student learning for racially and ethnically minoritized students. Thus, attention to equity in HIPs needs to occur at every stage, including the outcomes and impacts of the program (Finley, 2019; Trogden et al., 2022). An equity-minded approach to HIP outcomes could also include institutional agents’ (e.g., faculty, staff) practices, structures (e.g., policies), and culture as the focus of change, rather than solely putting the onus of change on students (Bensimon, 2007; Patel, 2015). Thus, in bringing an equity-minded lens to programs intending to broaden participation in undergraduate US STEM education, our framework brings together a critical examination to HIPs prominence, design, implementation, and outcomes.

Methodology

A systematic review is a useful method to identify themes and gaps in extant literature (Littell et al., 2008) and produce new knowledge, and thus aligns with the critical stance we take in this study (Mertens, 2012; Suri, 2013). Specifically, we examined publicly available multimodal data. Multimodality is both a phenomenon to be studied, and a method of studying the phenomenon (Kress, 2009). The phenomenon is the way we communicate using and integrating a variety of meaning-making resources, or modes (Kress, 2009). Multiple modes enable participants to convey meanings that are either complementary or coordinated (Martinec & Salway, 2005). The method should consequently draw systematically from the material conveyed in each mode, but then (re)integrate the modal material in the analysis.

We modified Petticrew and Roberts’s (2006) method of review for the social sciences, typically used for text, to allow for incorporation of various stakeholder voices through multimodal videos. The steps with modification include the following: (1) develop research questions; (2) select search terms and databases—in this case, focusing on the multiple modes of data available from each database; (3) formulate inclusion and exclusion criteria to refine the search; (4) assess study quality—in this case, determining how the databases have already screened for quality; and (5) extract data from the multiple modes into one format (text) for thematic synthesis using our conceptual framework to answer the research questions.

This is an innovative method distinct from content analysis, which scholars have applied to multimodal data (Serafini & Reid, 2019) and to systematic reviews in specific fields (Mikkonen & Kääriäinen, 2019). The main differences between this multimodal systematic review approach and content analysis are the epistemological stance and analytical steps. Content analysis is often interpretivist and uses open coding to split data into meaningful units (Mikkonen & Kääriäinen, 2019). We took a critical stance and used predefined categories derived from our HIP conceptual framework, as we describe further in the analysis section.

Search process

We used the STEM for All Multiplex (https://multiplex.videohall.com/) as the source for our multimodal data. The Multiplex platform serves as a repository that compiles the videos and related information (abstracts, discussion boards) from the annual STEM for all Video Showcases, from 2015 through 2022. Each Video Showcase featured federally funded programs designed to improve STEM and CS education from Pre-K through graduate school. Presenters created three-minute videos and abstracts of their programs with links to resources. During each eight-day Video Showcase event, viewers and presenters discussed each presentation through an online discussion board linked to each video.

The Multiplex is an appropriate database for the purpose of this study because it focuses on improving STEM education and offers publicly available, multimodal data (including video, abstract, and discussions) on each program in one database. The Multiplex contains 1620 project videos, of which 1430 of these projects are funded by NSF representing 56 different NSF programs. We chose to focus on NSF-funded programs because broadening participation and racial equity in postsecondary education is an increasing investment area for the NSF (James & Singer, 2016). The Multiplex and the related STEM for All Video Showcases have been viewed by 625,921 users who generated over 1.6 million page views. (Refer to Falk et al., 2019; Ives et al., 2022 for more information on the Video Showcase and Multiplex).

We screened the 1430 NSF-funded program videos featured in the Multiplex using the following criteria. The program needed to focus on (a) improving the STEM experience for undergraduate students, (b) four-year institutions, and (c) broadening participation in STEM for underrepresented students as defined by NSF. We excluded higher education teacher training programs that related to broadening participation in elementary and secondary settings. We chose to focus on four-year institutions as the primary awardee (allowing for community college and other partnerships). All Multiplex projects represented grants to a prime institution in the US. Efforts to reform undergraduate education in other countries may differ due to structure, policy, and demographic representation. Despite the US focus of included programs, there was international interest as visitors to the site came from over 200 countries and territories.

Beginning with the corpus of 1430 NSF-funded project videos, the screening process (Fig. 2) left us with 38 included videos for our systematic review. The first round of screening included tags and categories generated in the database for NSF-funded videos focused on improving STEM for undergraduate students, and for the second and third round of screening, we applied our inclusion criteria first to the video abstracts and then the videos themselves. The 38 included programs represent 5 of the 8 NSF directorates, plus two offices under the director.

Fig. 2
figure 2

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA)

Full size image

While the included programs may not be representative of all efforts to broaden participation in undergraduate STEM education, or the most effective at reforming student learning (James & Singer, 2016), NSF is a powerful funding mechanism that shapes STEM programs across the USA. Therefore, it is an important source and can help us examine the extent to which HIP prominence, design, implementation, and outcomes are equitable, if at all.

Data sources

For the systematic review of multimodal data, we examined primarily visual (text, images) and auditory (spoken word) modes (Pauwels, 2012). Data on the 38 included programs extracted from the Multiplex included the following: (a) 3-minute videos, designed for a broad audience; (b) written abstracts accompanying the video, short texts written for a broad audience; and (c) asynchronous discussion board posts where presenters responded to comments, questions, and feedback on their videos, offering additional insight and context into their program. From the NSF website, we collected the included programs’ NSF-submitted abstracts. These are longer texts written for a research audience, and often include details on goals, theories, methods, and outcomes.

After data collection, we sent a survey to the presenters of included programs. The survey served two purposes: (1) to confirm the validity of our data with video presenters (member-check) and (2) to triangulate our data sources. The survey included quantitative and qualitative questions focused to elicit which HIPs were included in their programs, key implementation characteristics of those HIPs, and the impact of their program. We compared the survey responses to the data we collected for each program from other sources to check for any convergent or divergent data (Campbell et al., 2020). The survey had a 33% response rate and confirmed our systematic review data.

Data analysis

The last step of data preparation was to represent the data from all sources in a common format (text) for synthesis to answer the research questions. We did this by creating a matrix based on our research questions and conceptual framework. The columns allowed us to include deductive categories and synthesize the data across all databases (Multiplex, NSF, and survey), while being open to inductive themes. We tracked the source of each data point by using color-coded text. The color coding allowed us to triangulate the data across sources. Matrices enable researchers to systematically display large data sets to draw conclusions about larger phenomena related to the research question (Miles et al., 2019).

Results

We organized the presentation of results according to our research questions, which focused on (1) the prominence of HIPs in the program, (2) the equity-minded design characteristics used by each project, (3) the implementation characteristics for each project, and (4) and the outcomes being measured to determine impact of the programs.

What HIPs are prominent?

Of the 38 programs, over one-third of the programs focused on a single HIP (37%), over a third included two HIPs (37%), a smaller group had three HIPs (16%), and only a few programs had four or five HIPs (5% each; Table 4). Most programs focused on the HIP undergraduate research (79%). The second most prominent HIP was internships (32%), often focused on research. Six other HIPs were included in a moderate number of programs each, while service learning, writing-intensive courses, and ePortfolios appeared in 1–2 programs each.

Table 4 HIP prominence
Full size table

Some programs had practices that were not included in Kuh’s list of 11 HIPs but are considered elsewhere in the literature and encouraged by specific requests for proposals. Five programs (13%) included interdisciplinary research or content as a key part of the program and student experience, and 16 programs (42%) noted the importance of partnerships or alliances between institutions, or between institutions and organizations or corporations.

  1. 2.

    How were HIPs designed to ensure equity-mindedness, if at all?

Most programs included equity-minded design characteristics (79%). Almost half of the programs included a single equity-minded design characteristic (42%), a smaller number included two to three equity-minded design characteristics (18% and 13%, respectively), and only one program had four and one program had five equity-minded design characteristics (3% each).

More than half of the programs (53%) provided increased resources to students such as scholarships, stipends, and paid work. A moderate number of programs provided resources and knowledge to change agents through professional development (37%). Eleven programs (29%) mentioned naming and addressing hidden barriers to inequity, such as specific courses, financial and work constraints, and finding mentors with similar identities. Fewer programs (16%) mentioned modified and tailored approaches, such as virtual or hybrid options, which was often due to COVID-19. While allowing more equitable access to HIPs, it is unclear whether this was the intent of virtual offerings or if formats will revert to in-person. Last, only two programs (5%) accounted for intersecting systems of power, when noting how participants had multiple oppressed identities and experienced multiple barriers that needed to be addressed.

As for the characteristic, designed with underrepresented students in mind, most programs had broad statements about intended student population (e.g., “underrepresented minority,” “underrepresented backgrounds,” “diverse backgrounds,” or “traditionally underrepresented” groups). Only 10 programs (26%) focused on specific racial and ethnic minority groups. These programs focused on Black, Latinx, Native American, and Pacific Islander students. Interestingly, survey respondents who stated that their programs were meant to serve racially or ethnically minoritized students broadly also reported a high proportion of White students in their programs. For example, when asked the demographic breakdown of their programs in the survey, some (4) said that their programs comprised 42–75% White students.

In pursuit of equity, funders have been increasingly encouraging multi-institution proposals (Dopke & Crawley, 2013). Of the 38 programs included in this review, 14 (37%) included collaboration across multiple institutions. As for institutional types represented among the programs, over half (61%) included a minority-serving institution as the sole primary or partner site (specifically, Hispanic-serving institutions, historically Black colleges and universities, and Tribal colleges and universities).

What are the implementation characteristics of the HIPs reported by the study projects?

Programs varied in their description of implementation characteristics. Most programs (87%) included interactions with faculty and peers about substantive matters. Over half of the programs used peers to mentor students in the program, 40% used staff to mentor students, and over 90% used faculty to mentor students. While interactions with faculty and peers about substantive matters was the most frequent implementation characteristic, over a third of survey respondents identified peer interactions as a challenge or implementation failure, and one program identified faculty mentoring as a challenge. For example, one principal investigator said in the survey that not all faculty followed through with the mentoring program.

Twenty-nine (76%) programs noted a significant investment of concentrated effort by students over an extended period. Over a third of programs included public demonstration of competence such as a conference presentation (42%) and setting high performance expectations such as GPA and tutoring requirements (37%). A smaller number of programs addressed real-world application of learning (21%), autonomy and accomplishment (18%), and structured opportunity to reflect on learning (16%). Less frequently, three programs (8%) focused on students making a difference in their communities, two programs (5%) included international collaborations for exposure to diversity, and one program (3%) mentioned constructive feedback.

What outcomes are being measured to determine impact?

Many of the 38 programs used conventional measures for determining outcomes and impact that focus on how the program added value to the students’ learning and development. For example, programs used retention/persistence, graduation/completion, or transfer (from 2- to 4-year college) rates (45%), and/or cognitive measures such as grades, knowledge (26%) to illustrate impact. About one-third of programs included psychosocial measures (34%), such as academic identity, motivation, anxiety, and self-efficacy and a quarter of programs included sociocultural measures (24%) that relate to broader cultural transformation, such as a sense of belonging and support. Fewer programs (18%) used a control group to illustrate student success as compared to a comparable non-participant group. While the inclusion of psychosocial and sociocultural measures indicates progress in the field, these are often based in students’ perceptions of institutional culture change, rather than actual institutional transformation. Only 4 programs (11%) included measures related to transferability or scalability of their program as a model for others, changing systems and structures within the institution.

An equity-minded approach, however, necessitates shifting the focus of outcomes from measuring changes in the student, to measuring changes in the institution or institutional agents (Bensimon, 2007) that align with the HIPs included in the program. For instance, one program used a research project that helped students identify “barriers in university culture” and provided researchers “insight into institutional cultures and norms that undermine[d] the success of URM students.” These qualitative outcomes align with the equity-minded design characteristics of naming and addressing hidden barriers. Yet, programs could go further in measuring and identifying impact in areas such as changing policies, resources, and practices to be more equitable.

Discussion and implications

Our conceptual framework for this study comprised (a) 11 high-impact practices, (b) 6 equity-minded design characteristics, (c) 10 implementation characteristics, and (d) outcome measures (Fig. 1). We summarize each finding and situate it in the literature before providing implications for researchers and funders. We conclude the discussion by synthesizing across findings from our conceptual framework, highlighting two exemplar programs. The exemplars are not meant to be the ideal, but rather bring the data to life and illustrate how the conceptual framework can be used as a model to develop and evaluate HIPs.

Systematic integration of multiple HIPs: maximizing variety

We found most NSF-funded programs in this study included one to two HIPs, with undergraduate research being most frequently adopted. Only four of the 38 programs integrated four or more HIPs. We also found two program features emerged inductively—alliances or partnerships, and interdisciplinary learning. The inclusion of these two features is perhaps shaped by requests for proposals (RFPs), as certain NSF programs emphasize partnerships and interdisciplinarity (Holley, 2009). However, these features could be further explored to determine impact on broadening participation in STEM.

The lack of variety in HIPs and the limitation of the number of HIPs integrated into a single program could be hindering broadening participation in STEM. Students’ experiences improved when participating in more than one HIP (Finley & McNair, 2013), and racially minoritized students often face limited access to HIPs (Finley & McNair, 2013; Greenman et al., 2022; Kinzie et al., 2020). Therefore, while undergraduate research and internships are important discipline-based HIPs, they may be insufficient when not infused with other HIPs. Systematically integrating other HIPs into undergraduate research and internships can provide greater access to important experiences, skills, and outcomes embedded in HIPs that racially minoritized students are less likely to experience (e.g., Loker & Wolf, 2022).

Consequently, we recommend that researchers and funders consider maximizing the number and variety of HIP offerings within funded programs. Faculty and administrators could map the HIPs offered at their institution, considering how different HIPs align with the priorities of departments (Trogden et al., 2022), and query which HIPs are missing and who has access to STEM-focused HIPs. Rather than trying to “add-on” HIPs into already existing programs, such as placing underrepresented students in already-established faculty research labs, researchers and funders could systematically integrate HIPs into teaching and service requirements (Whittaker & Montgomery, 2014). (Refer to Wilson et al., 2022 for an example of a professional development program incentivizing faculty to incorporate HIPs into courses). Integrating HIPs into classroom experiences could increase access for racially minoritized students. The literature also indicates that interdisciplinarity can enhance student learning outcomes (Ivanitskaya et al., 2002) and has the potential to transform institutional culture (Holley, 2009) as stakeholders work across traditional silos.

Equity-minded design characteristics: targeted programs

The findings show that most NSF-funded programs included only a few equity-minded design characteristics, such as providing additional resources to students and providing resources and knowledge to change agents through professional development. Further, a quarter of programs targeted specific racially and ethnically minoritized groups, and half included a minority-serving institution as the main or collaborating research site. Scholars have found it is important to design HIPs with underrepresented students in mind—hearing their voices and understanding what they view as high impact (Harper, 2010; Kinzie et al., 2021) with attention to within group differences (Stewart, 2013). This begs the question, are programs for underrepresented STEM students as a collective targeted enough? Trogden et al. (2022) emphasize the importance of disaggregating student data to examine equity gaps and targeting HIPs to specific minoritized populations to eliminate those gaps. Using equity as a linchpin when planning HIPs should be the overall goal so that they are accessible and beneficial for minoritized groups (Zilvinskis et al., 2022). If HIPs are designed with the needs of particular minoritized students in mind, HIPs will become truly inclusive and, in the end, benefit all students (Stewart & Nicolazzo, 2018).

While funders are increasingly focusing on equity and making strides in increasing funding to minority-serving institutions, McCambly and Colyvas (2022) found that grantmaking often reproduces the racial status quo. Funders can up the ante on equity-minded design in RFPs. For example, as a starting point, NSF has developed specific calls for minority-serving institutions and racial equity in STEM education. Program officers can also think about equity-minded design when reviewing proposal conceptual frameworks and theories of change.

Implementation characteristics: training change agents and missed opportunities

When researchers and administrators implemented HIPs, we found characteristics varied, with most programs including peer and faculty interactions, ranging from faculty mentoring to peer mentoring networks. Scholars emphasize the importance of faculty and peer relationships in underrepresented student success (Kinzie et al., 2021). However, in their study on racially minoritized first-year and senior-year students in HIPs, Kinzie et al. (2021) found that “although substantive interactions are a key component of HIPs . . . some of these interactions can be harmful” (p. 11). While an analysis of the content and quality of interactions was beyond the scope of this study, we note that over a third of programs incorporated a key equity-minded design feature of providing agents of change (faculty and staff) with the knowledge and resources to implement change through professional development (PD). This PD could be critical in improving interactions. However, we did not have data to determine how equity-minded the PD workshops, in fact, were. This is a key area for researchers and funders to consider as equity-minded change, if it is to be sustainable and persistent, must address implicit beliefs, attitudes, assumptions, and practices of faculty and staff, rather than solely focusing on student interventions (Bensimon, 2007). Bartell and Boswell (2022) provide an example of how “high-impact faculty development” can moderate the impact of HIPs on racially minoritized students (p. 50).

Our framework included two recently added implementation characteristics (Kinzie et al., 2021)—students making a difference for others, and autonomy and accomplishment. There is a wealth of research on the importance of considering minoritized students’ lived experiences and cultural knowledge in the classroom (e.g., Castillo-Montoya, 2019; Rios-Aguilar, 2011). Yet only 21% of programs in this study considered real-world application of learning, 18% gave students a sense of autonomy and accomplishment, and a mere 8% emphasized making a difference in their community. This lack of connection to students’ lives and communities, and giving them the power to make those connections, is a missed opportunity to attract and retain underrepresented students in STEM.

Outcomes and impacts

Our last research question focused on the outcomes and impacts of HIPs. We found conventional outcome measures, such as retention/persistence, graduation/completion, transfer, and employment or graduate school entrance rates, and cognitive measures such as grades, content knowledge, sense of belonging, and self-efficacy. As for the broader impact on society, the overarching goal of all 38 programs was broadening participation. However, some programs also emphasized transferability or scalability of their program as a model for others, which is a key aspect of NSF funding. The relatively rare attestation of this characteristic may be attributable in part to the early stage of development of programs.

While measures of impact are often quantitative, the Video Showcase with its multimodal affordances enabled participants to illustrate impact qualitatively as well (e.g., with audio or video clips of students participating in or discussing the outcomes of the program). The Video Showcase discussion also allowed researchers to share common challenges, struggles, or strategies of implementing HIPs and measuring outcomes and impact. Innovative ways to approach measuring outcomes and impact can help shape future design of HIPs. Therefore, we suggest researchers participate in these types of multimodal networking events, as well as agencies to fund these events and professional development opportunities (Ives et al., 2022).

Further, Whittaker and Montgomery (2014) suggest funding agencies should “move beyond providing financial support” to requiring assessment, sustainability, and institutional accountability (p. 270). We suggest funding agencies consider providing training and transparency on how broader impacts can be tracked for accountability. Furthermore, alliances have the potential to broaden impact and sustainability, as the program is tested in multiple institutional contexts. While there has been movement at NSF to integrate racial equity in broadening participation, efforts to change outcome measures and impacts lag. Funding agencies can critically examine equity across all broader impacts, rather than siloing equity within broadening participation.

Program exemplars

These two exemplar cases were selected because they highlight multiple aspects across the framework working together and draw on various points of dataFootnote 2. The first case shows how a program can include multiple HIPs, as well as equity-minded design and implementation characteristics in tandem. The second example shows how a program can include multiple HIPs and move beyond usual measures of outcome and impact. These two exemplars illustrate how the study framework can be used as a reflective tool for researchers and practitioners as they think about designing and carrying out HIPs.

The first exemplar is of a 4-year university, community college, and research center partnership that offered experiential learning opportunities in the form of virtual internships to non-traditional, under-represented, adult learners. The goal of the project was to provide this student population with the opportunity to gain professional skills that employers in STEM are seeking in graduates. This program creatively combined three HIPs (Table 5), as students in courses, including capstone courses, are placed in teams to complete an employer-sponsored project. The industry partner sponsoring the project provides feedback to the students and instructor through the virtual platform, which also scaffolds professional skill development by organizing project milestones, collaboration, and deliverables.

Table 5 Exemplar #1
Full size table

This program drew on several of the equity-minded design characteristics. For example, the virtual and free internship platform was a modified and tailored approach that opened access to those who often face barriers to participating in experiential learning. As for implementation, the program offered students real-world experience, and opportunities for employer feedback and reflection were integrated into the virtual platform. Further, students collaborated with peers on the sponsored projects and interacted with their instructor through the platform. While the internship experience, integrated within a course, lasted the full semester, institutions could adopt and integrate the platform into multiple courses over multiple years. Further, the project team offered professional development onboarding to new college partners to enhance their knowledge of the program before implementation (increasing knowledge and resources of change agents).

While the project was already underway when the COVID-19 hit, the pandemic allowed them to focus on implementation across multiple contexts. The team documented conventional outcome measures, such as student employment and confidence in their skills. However, they also focused on scalability and transferability as they expanded to 16 participating institutions. The researchers employed design-based research to examine how the virtual internship model was taken up and adapted to each institutional context and whether this innovation was sustained.

The second exemplar is of a program focused on increasing STEM participation of low-income students by offering participants a four-year scholarship. While the overall goal was to increase the success of students with demonstrated financial need, the program also included objectives to ensure scholars were first-generation college students or racially/ethnically minoritized. The program included five HIPs (Table 6). While the program did not include many equity-minded design characteristics, they did offer four years of scholarship money for students as a resource. As for implementation characteristics, this four-year program included instances of high effort over time. It also had high expectations of students, evidenced by providing students with multiple forms of academic support. Furthermore, they included peer interactions through a cohort model.

Table 6 Exemplar #2
Full size table

The program used conventional measures for determining outcomes and impact but moved beyond that to include measures that aligned with the HIPs under study. In the survey, the principal investigator indicated measuring the following outcomes: (a) retention or persistence rates, (b) graduation or completion rates, (c) grades or GPA (measures of content knowledge), and (d) cognitive or sociocultural measures (self-efficacy, sense of belonging, identity). An example of aligning outcome with HIP characteristics includes measuring students’ sense of belonging and confidence in reaching out to faculty. For quantitative measures, the researchers used a control group. As noted in a discussion board post, “Scholars more strongly identify as STEM professionals than control students, although not significantly so. Scholars more strongly feel as if they belong in the science community than control students, at a significant level.” These outcomes are representative of the HIP effort to build community and a sense of belonging for students through a first-year experience, common intellectual experiences, and a learning community. As this program only received its funding two years before presenting in the Video Showcase, they did not include significant data for outcomes measured. However, they did note desired impacts by setting a benchmark of achieving a 90% 4-year graduate rate and 85% STEM degree completion rate. These quantitative measures can be tied to the more skill-focused HIPs (writing courses and research), which can aid students in gaining the skills they need to succeed. These two programs offer examples of how it is integral to consider equity when designing HIPs, and then consider how to tie outcome measurements to the impact of the HIPs.

Limitations

We acknowledge that no systematic review is exhaustive. However, the purpose of a systematic review is not to exhaust the literature, but rather to synthesize ideas and methods in the field within certain parameters. While we included all relevant sources pertaining to our research questions, our study still has a limited scope. We examined a subset of NSF-funded programs, as not all programs participate in the Video Showcase. We also acknowledge other federal and state funding sources also aim to broaden participation in STEM higher education not captured in this review. Our review features programs that were successful in responding to NSF RFPs, and thus they may have some normative aspects of program design that were responsive to the RFP rather than aiming for equitable transformation. Therefore, the included programs may have qualitatively different elements than non-NSF and non-participating programs. Lastly, for the programs included in the review, we relied on video, abstracts, discussion, and surveys as data sources. The program may have included HIPs not highlighted in these data sources, or varied in quality of implementation that was difficult to measure from the data sources.

Conclusion

Scholars call for the field to critically examine HIPs through an equity-minded lens. How are researchers and practitioners choosing which HIPs to include in their programs, and designing them with equity in mind? How are they then implementing—or carrying out—those HIPs, and further, how are they measuring the impact of the HIPs in ways that reflect the intention to broaden participation in STEM? This paper provides a framework that practitioners, researchers, and funders can draw on to examine how HIP design, implementation, and outcomes can dovetail to create a dynamic program to advance equity in STEM for racially and ethnically minoritized students. Further, this systematic review provides an overview of the field of HIPs that is difficult to recognize when looking at a single program or institution. While we found the field is moving the needle in using federally funded programs to broaden participation in STEM through HIPs, we also found places where we are falling short. As federal funding is a key mechanism by which institutions can implement HIPs, we call for researchers, practitioners, and funders to maximize the variety of HIPs included in each program, target the programs to specific underrepresented groups, connect to students’ lives and communities, and consider equity-minded impact measures.