Introduction

A diverse workforce trained in dissemination & implementation (D&I) science is critical for improving cancer outcomes and reducing cancer-related health disparities. D&I science is the study of methods to promote the systematic uptake of evidence-based interventions and practices to real world context to prevent diseases or reduce their burden [1]. Training of students, public health professionals, and researchers is an important strategy to expand this workforce. Over the past two decades, there have been numerous efforts to increase training opportunities in D&I science through workshops, institutes, and degree-seeking programs. The National Institutes of Health’s Training in Dissemination and Implementation Research in Health (TIDIRH) began as a 5-day institute including a balance of structured content topic areas, interactive small-group discussions, and personalized sessions on individual trainee projects [2]. The ultimate goal of the institute was to increase the submission rate and quality of D&I grant applications and scholarly publications among trainees. In later iterations, it has been delivered virtually and an adaptation of it was developed for professionals who worked in cancer control [3]. Additionally, institutions with Clinical Translational Science Awards (CTSAs) have focused on clinical translational science and training; they also have emphasized the importance of D&I science methods to translational science. In one survey, about 70% of CTSA leaders reported they directly or indirectly supported D&I science training; however, they also identified challenges to D&I activities at their institutions, such as funding, limited understanding of D&I science, and a small D&I workforce [4]. Furthermore, schools of public health and other medical institutions have developed degree programs with many at the doctoral or post-doctoral (T32) training in this area [5].

Most existing training opportunities have enrollment limits and many have focused on advancing the knowledge and skills of academic faculty researchers or clinical investigators; thus, there is a D&I training gap for early-stage researchers, such as, graduate students and post-doctoral fellows, and for practitioners employed in healthcare and community-based public health settings. The Cancer Prevention and Control Research Network (CPCRN) was established in 2002 and is a Centers for Disease Control and Prevention (CDC) funded network of eight centers whose mission is to implement evidence-based cancer prevention and control strategies in communities. The CPCRN Scholars program was launched in 2021 to build capacity by nationally scaling available training (e.g., Putting Public Health Evidence in Action-PPHEIA) across all CPCRN sites and beyond, and mentoring students, researchers, and practitioners in D&I science for cancer prevention and control [6]. There are existing D&I competencies that were used to measure scholars’ understanding of D&I science and informed program development. The CPCRN Scholars program uses a combination of teaching modalities: asynchronous digital content for self-paced, self-study; synchronous webinars and small-group discussion; and individualized mentorship on a scholarly project. The program curriculum emphasizes evidence-based public health/cancer interventions and D&I scientific frameworks, strategies, and methods to advance knowledge and increase competencies related to implementation research and practice. Students completed one of two D&I curricula, the Putting Public Health Evidence into Practice Training or NCI’s Training Institute of Dissemination and Implementation Research in Cancer, open access modules [3, 7,8,9]. They also were invited to learn about the CPCRN and related D&I science through readings, attendance at the annual meeting, network-wide webinars, and workgroup participation [10].

In this paper, we describe the program and the evaluation conducted to test the program’s impact in preparing scholars for collaborative careers in cancer control and D&I science. We also offer recommendations for program improvements guided by evaluation results.

Methods

The CPCRN Scholars Cross-Center Workgroup (WG) was formed in 2019 to develop and deliver a D&I Scholars training program, leveraging the resources, infrastructure and expertise of the CPCRN. The WG first delivered the program to 20 scholars in 2021. To evaluate the inaugural program, we conducted a sequential (quantitative + qualitative), mixed methods study [11]. All activities were coordinated by a CPCRN Scholars WG and led by two CPCRN principal investigators and program directors (University of South Carolina and Emory University) and included members of the eight CPCRN centers. The locations of the 8 CPCRN centers are shown on the map in Fig. 1. The University of South Carolina IRB approved the evaluation study.

Fig. 1
figure 1

The Eight CPCRN Centers Across the United States

Full size image

We conducted a baseline survey with all scholars after they volunteered to participate in the evaluation and prior to initiating program training activities in January 2021. At the conclusion of the program in Fall 2021, all scholars were invited to complete a follow-up survey [Supplemental Fig. 1]. We then invited all Scholars and ten of their mentors to participate in 30-min interviews from September to November 2021. Four interviewers completed the interview after verbal consent was given.

The survey included demographic questions and information about their affiliation either their current educational program or organization, in addition to assessing the baseline metrics of D&I competencies. The exit survey asked about overall program evaluation, and attitudes about the program. Scholars were asked to assess their D&I science competencies on a scale of 1-beginner to 3-advanced, as they did at baseline [12]. The 48 competencies encompassed 4 broad domains: (1) definition, background and rationale, (2) theory and approaches, (3) design and analysis, and (4) practice-based considerations. Scholars also rated their satisfaction with each program component, mentoring and the overall program. The final items included questions about use of the Scholar’s program materials in the future, applying D&I science concepts in research and practice, collaborating with fellow CPCRN scholars, and recommendation of the program to others.

An evaluation subgroup of the Scholar’s Workgroup designed the semi-structured guide. A program director administered the survey and sent reminders. The sample included all of the scholars and pre-selected mentors (n = 10) to facilitate rapid data collection once the program was complete. The Scholars Program leadership team compiled a list of 15 actively engaged Scholar program mentors whom represented diverse D&I science and geographic regions to participate in the interview. Mentors with availability were interviewed. The exact number of mentors engaged with scholars across the program is unknown due to scholars changing mentors throughout the program and the practice of garnering informal mentorship through workgroup projects. The interview guide included 14 items that covered their overall program experience, reasons for joining the program, description of their project and working with their mentor(s), opinions of the training and webinars, and how the program enhanced their cancer control research, and/or understanding of D&I science. Finally, the guide asked about benefits of the program, suggestions for improvements, and how the program helped them with their career goals. The interview guide for the mentors was composed of 19 questions that covered three domains: mentor experience, Scholar’s project, and the program in general.

The surveys were programmed into Qualtrics [13]. Data were downloaded into SPSS version 26 for analysis [14], and merged for the scholars’ baseline and follow-up data. Descriptive statistics including frequencies and means were run and scales were computed. Data were stratified into two groups (students and practitioner/researchers) in the analyses. For the qualitative analyses, a cancer center’s shared resource conducted the analyses using analyses methods such as reiterative codebook refinement and dual coders [15]. A codebook was developed based deductively on the interview guide; however, other inductive codes were created based on the interview transcript discussions [16]. Two staff independently coded the data based on codebook definitions and met about any discrepancies to reach consensus, shared the coding with the program leads and then conducted thematic analyses of the themes using NVivo 12.0 [17]. Data matrices of the codes were created for the scholars and mentors, and representative quotes were selected by program leads.

Results

Baseline and post-program survey findings

All scholars (n = 20) completed the baseline survey and 17 completed the exit survey. For the interviews, 18 scholars (90% response rate) and 10 mentors (50%; percentage based on every scholar having one mentor; total number of mentors is unknown) completed the interviews. For the inaugural year, 50% of trainees were students, 45% researchers, and 5% practitioner scholars (Table 1). Scholars overall were 83.3% female and 66.7% White, 27.8% Asian, and 5.5% Hispanic. Nine scholars were students who were at the doctorate level (50%) and from fields such as decision science and health outcomes, epidemiology, and health promotion.

Table 1 Demographics of the Scholars (N = 18)
Full size table

Competencies Related to the D&I competencies, scholars’ ratings of their competencies all increased from baseline to posttest (Table 2). Some of the competencies with the greatest change included identifying and applying techniques for stakeholder analysis/engagement (delta, d = 0.65), describing relationships between organizational dimensions and D&I research (d = 0.54), identifying a process for adapting an evidence-based intervention (d = 0.53), and using evidence to evaluate and adapt D&I strategies for specific populations, context, etc. (d = 0.48) Although there was not a significant increase in the total D&I competency score (M = 4.39 vs M = 4.56), the scholars reported significant increases from baseline to post-program on all of the subdomains (data not shown). The subdomains that reported the largest increases from the baseline survey were practice-based considerations and D&I design and analyses.

Table 2 Rating of Dissemination and Implementation Science Competencies by Scholars’ Group
Full size table

Engagement Related to the curriculum, 94.4% reported attending the PPHEIA webinars and 77.8% reported completing the NCI Dissemination and Implementation modules (Table 3). About 33% completed their project, while 55.6% partially completed it. A large majority (94%) reported receipt of mentoring or partial mentoring in the program. About 78% of scholars engaged with a CPCRN workgroup. The program components that were ranked most valuable (defined as very or moderately valuable) were the PPHEIA training (88.9%), NCI D&I training modules (83.3%), and webinars (kickoff webinar-88.9% and selecting implementation theories and models-88.9%). Two other components were rated moderately valuable: mentoring (72.2%) and engagement with CPCRN project or workgroup (72.2%).

Table 3 Overall Scholars Program Evaluation (N− = 18)
Full size table

Satisfaction For overall ratings of the program, 77.8% reported satisfaction with the format of the activities and ability to speak of D&I concepts (Table 4). Overwhelmingly, trainees agreed that knowledge about the CPCRN increased (88.9%). About 83% reported being very or somewhat satisfied with the mentoring received, 72.2% reported strongly agreed or agreed that their mentor helped engage them with a workgroup, and 66.6% reported that their mentor expanded their network. All scholars (100%) reported that they would use the curriculum materials in the future and would recommend the program to others.

Table 4 Satisfaction with Program and Learning Transfer (N = 18)
Full size table

Post-program interviews with scholars

Scholars reported a variety of benefits of the program. Scholars reported that learning implementation strategies, frameworks, and methods was useful (n = 6). One scholar stated, “So it's introduction to implementation science, theory, methods, and frameworks, implementation strategies. So these are also very useful. But it's always more interesting at the—at least for me it's very interesting to hear from people actually doing projects on the ground because it helps you.” Some scholars (n = 4) noted that the program’s variety of teaching methods and mediums such as using websites, PowerPoints, modules, and a variety of other tools, were very beneficial aspects of the program. Some participants (n = 3) found that the curriculum helped with their work outside of the program as a teaching assistant, or through a class assignment, dissertation, or another project.

Scholars were asked to share their experiences in the CPCRN Scholars program. Several scholars (n = 4) shared that due to the CPCRN program they have more knowledge about D&I research: “I think one of the things that I got the most out of it was just the educational experience, getting a better grasp and understanding of D&I research and supporting building a foundation to it. That’s probably what I got the most out of it, was just through the webinars and, in the readings, became much more, I guess, knowledgeable about D&I research. That’s probably what I benefitted or enjoyed the most out of it.” Participants shared how the program was beneficial for their careers (n = 4), for their own research (n = 2), and for building their network (n = 2). Regarding how the program benefited one’s career, one scholar shared: “I think it was a nice addition to what my work has been as a project director. I feel like there was a fair amount of overlap, and it allowed me to really be integrated into the network, and it also allowed me to have a better understanding with D&I science and its scope and its range since that's not my area of study currently…”.

We assessed the scholars’ perspectives on the workgroup opportunities in the program and their involvement in workgroups. Scholars reported a positive aspect of the workgroups was the opportunity the workgroup provided to meet others, network with people, and collaborate with other scholars on projects (n = 3). For example, one scholar described how she was able to collaborate with another scholar from a different institution on a K award (an award funded by the National Institutes of Health (NIH) to provide support to junior faculty on their research) due to shared research interests and both their expertise complemented each other. A few scholars described workgroups as inviting and reported workgroups had an inclusive, positive culture (n = 2). Some workgroups met more frequently than others. Scholars involved with workgroups who met frequently found that helpful. For example, one scholar stated: “There is a biweekly meeting that we had with a smaller health equity group just to keep the project going. So that was very helpful.” In workgroups, scholars were able to get feedback from collaborators outside of their institution (n = 2) and they gained an understanding of the structure and inner workings of the workgroup and its projects (n = 2). One scholar stated: “And people would seek broader opinion from the group, and that is how I had an experience getting feedback from the other collaborators who are not associated with the institution. So this is again a great learning experience, especially at a student level. So it is my first time working with so many collaborators from different universities.” One major disadvantage that participants (n = 5) noted was not joining a workgroup during their time in the program, and other challenges to workgroup engagement included having trouble joining a workgroup (n = 1), workgroups conflicting with their class schedule (n = 1), and workgroups having infrequent meetings (n = 1). For example, one scholar stated: “I tried the organizational readiness one. But I think only one of the meetings actually happened. The rest of them were canceled. And the one that happened I was in an all-day board meeting. So I couldn’t attend.”

Other valuable program components were mentioned. One positive effect of the program on the scholars' networks was that it provided access to future collaborators on papers and post-doctoral positions (n = 3). One participant stated: “I have made connections with a couple of faculties to get into my PhD research work. So yes, I have made connections, and it was very helpful networking with people through my participation as a scholar in this program.” Select scholars (n = 3) mentioned that the program helped them expand connections through access to the established CPCRN network or professionals. Other ways the CPCRN program impacted the scholars’ networks included facilitating connections with different academic centers (n = 2), the ability to make connections through the workgroups (n = 2), and getting to know people outside of the scholar’s home institution (n = 2). Table 5 presents a joint display of some evaluation results with the program’s quantitative ratings and exemplar quotes from different types of scholars.

Table 5 Joint Analyses of Survey and Interviews of CPCRN Scholars
Full size table

Post-program interviews with mentors

Mentors were asked questions about the CPCRN Scholars program, including their general impressions, challenges, purpose of the program, and the most important area of the program. Mentors shared that interactions with scholars happened during individual meetings generally consisting of discussion around the scholar’s project and training activities. Interactions were reported to be smooth and easy, with some discussions broadening beyond the CPCRN scholar training. Three mentors stated that they interacted with scholars at scheduled meetings, with one participant stating, “I think we built it into our weekly or bi-weekly meetings so she would let me know how many interviews she’s done, where she is on analysis.”

Mentors shared their thoughts about the most important aspects of the Scholars’ program. Three mentors mentioned this was networking, with one participant saying, “…the most impactful was getting people integrated into this larger network. And realizing—allowing them to realize the richness of experts across the nation who are working in D&I science that they can link into. I think that was the most impactful.” Four mentors discussed their general satisfaction with the program, with one saying “I think it has been a great program and a real benefit to the participants….” Three mentors mentioned that the program allows scholars to build relationships, with one explaining the importance of keeping scholars engaged; one commented, “I think having the program is a good opportunity to get people engaged in the work of the CPCRN. I liked that, at least it seemed like, from a worker perspective, you were trying to match scholars with these different workgroups to collaborate with other investigators. I think building those networks and relationships is really important to do. The sooner you can do them, the better. Yes, I think pretty favorably of the program.” Table 6 presents key findings related to mentors’ perceptions of the program.

Table 6 Interviews with Mentors in the Program
Full size table

This mixed methods evaluation also asked about recommendations for program improvements. Open-ended questions on the survey and an interview item were included to allow respondents to suggest programmatic changes. Recommended areas for improvements from scholars included greater opportunities for networking and mentoring, more communications around program expectations, and extending the length of the program or project completion timeline. For example, many students (n = 5) shared that there was little to no interaction with other scholars. One participant specifically shared, “I had a chance to interact with them {fellow CPCRN Scholars}, but not much. Just it would have been an introductory interaction kind of thing. But I was not able to continue interacting with them.” Similarly, mentors wanted more information about the program and expectations to help them with their roles.

Discussion

Overall, we found high ratings for the inaugural year of the Scholars Program across many program components. Scholars valued the curriculum, engagement with an established, national cancer research network and advising by their mentors. Generally, the scholars rated the two D&I curricula and engagement with the CPCRN higher than other program components. Their learning was evident in changes in D&I competencies across many topical domains. Related to mentoring, scholars who already had a strong working relationship with a current mentor or their mentor was already active in the CPCRN reported having the following: more engagement with their mentor, their mentor helped the scholar engage them with a workgroup, linked them to other researchers in their areas of interest, and a positive mentorship experience. Expansion of their network is critical since many of the scholars were students and this may build their relationship to the fields of D&I science and cancer control.

We found that scholars highly rated the fact that the training was in the context of an established national research network: learning about the history of the CPCRN, working within CPCRN topic-focused workgroups, and networking nationally with members located in geographically diverse settings as part of the program. The unique multi-institutional infrastructure of the CPCRN and its many topical workgroups around cancer, health equity, and health-related translational research projects provides many opportunities for the scholars for engagement and real-world application.

In turn, scholars catalyzed efforts and related outputs of the workgroups. For example, several scholars contributed to workgroup projects (e.g., equity checklist, data collection, literature reviews). This participation resulted in submission of abstracts and manuscript writing with a cohort of researchers and practitioners. Scholars who engaged in workgroups that met frequently reported a higher positive experience compared to scholars who were involved with workgroups who met on rare occasions or could not attend the meeting due to other commitments. Additional, scholars improved in their competency around contemporary D&I issues of health equity and community engagement (e.g., program adaptation, partnership engagement); researchers and practitioners have promoted these critical concepts and skills in practice [18,19,20]. Future evaluations (e.g., a planned scholar alumni survey) can explore the long-term impacts of this engagement for Scholar’s career development and continued engagement with the CPCRN.

The Scholars program fulfills the CPCRN goals to build the D&I workforce. Research has demonstrated the need for more D&I training and building capacity of various professionals across career stages [21, 22]. This program is unique since many D&I training programs are focused on researchers and it expands the trainees to include both students in public health and practitioners. In addition, to address health inequities, D&I scholars should be diverse and come from different setting; this program joins other national efforts to address health equity in translational science [23]. Future research could evaluate the program’s influence on the student scholars’ trajectory and how the program is helping to increase the pipeline of diverse scholars in D&I for cancer prevention and control and other outcomes specified in our program logic model [6].

The evaluation results from the inaugural year of implementation identified several areas for continuous process improvements. Scholars reported a desire for the program to be longer, with more interactions with each other. The flexibility of the program’s structure was greatly appreciated by all, however, explicitly communication about program roles and expectations was requested. For example, some of the tracks had extra meetings for check-in with the scholars while others did not. Expectations for mentor–mentee interactions could be clarified, as those frequently engaging found this valuable. Similarly, engagement in WG was reported to be important to program satisfaction. Importantly, the inaugural training program was launched during the international pandemic of COVID-19, thus limiting opportunities for face to face interactions which may have furthered engagement and possibly improved communications, but did allow for digital and later accessible webinars.

The inaugural year, during the height of the pandemic, served as a soft launch of the program. It was encouraging to see the number of represented institutions grew by seven, including one that is international, Swiss Tropical and Public Health Institute) in its second year (2021–2022 Scholars Cohort). The 2022–2023 Scholars applicant pool indicates continued strong growth with a 135% increase of applications from the second year and the 2022–2023 cohort consists of 21 schools and organizations globally. Programmatically, we had fewer practitioners who applied during the inaugural year of the program than in the second year of the program. Recruitment channels were expanded with partnering agencies and funders to increase awareness of the program among this group. The Scholars Workgroup made program modifications based on the evaluation results in extending the program by a few months (9 to 11), providing more information about the program and its components in advance of enrollment, establishing more frequent communications overall, providing additional information for mentors about the program and mentor expectations, and adding virtual Scholars mixers to the programming. The mixers were added for scholars to voluntarily share a few PowerPoint slides about their project aims and methods, gain input from other scholars and mentors and refine outputs and next steps. These additions were to enhance peer exchange and collaboration which have been identified as capacity-building mechanisms [24].

Strengths of this program evaluation study included use of mixed methods to assess opinions of the program and in-depth feedback about its components qualitatively. The data collection covered a breadth of topical domains for the evaluation from ratings of program components, D&I competencies, mentoring, and career developments. We included the mentors as part of the evaluation. However, the evaluation results may not be generalizable to other D&I trainees of students, researchers, or professionals as our training program has unique features. All data were based on self-report and information on progress on projects were not validated with the mentors. We interviewed a lead mentor for only half of the Scholars due to time and funding constraints. Future evaluations could include perspectives of all mentors.

Conclusion

The inaugural CPCRN Scholar program year yielded positive results, particularly related engaging with about the CPCRN and increasing knowledge about D&I science and cancer control among its student, researcher, and practitioner scholars. There were many reported benefits, largely centered on new professional relationships, connections and networking, and some suggestions for improvements. This training program can build the capacity of current and future researchers and practitioners in D&I science. Future research could explore the long-term impact of the CPCRN D&I scholar training and networking on career trajectories in cancer prevention and control as well as cancer health disparities research.