Background

Translational research aims to improve individual and population health outcomes by bringing evidence-based knowledge into clinical and public health practice [1, 2]. Health service researchers and clinicians are increasingly expected to not only produce high-quality research but also facilitate sustained implementation of knowledge in practice [3]. Medical schools and research institutions face increasing pressure to improve researchers’ and clinicians’ competencies in both knowledge production and knowledge translation [4]. Evaluation of the success of training programs is therefore a priority.

Training in translational research has thus far demonstrated positive results. Targeting graduate students for mentoring has been shown to influence graduate students’ career development trajectories. Long-term mentoring also encourages support of mentees to continue to evolve their academic research careers [5]. Many evaluations, however, emphasize academic productivity (in the form of publications and grants) as a primary measure of success [6,7,8]. While the skills of the research team or their record of interdisciplinary collaboration have served as additional outcome measures in a few programs [9, 10], there is an increasing need to establish relevant competencies of trainees for evaluating success at all levels of the knowledge translation process. Whereas several frameworks for assessing success of translational research efforts exist, no consensus has been reached on how to best evaluate these programs [11,12,13,14].

In this paper, the authors present an evaluation of a National Institute on Drug Abuse (NIDA)-funded translational research training program, the Institute for Translational Research Education in Adolescent Drug Abuse (“the Institute”), aimed at engaging graduate-level scholars and behavioral health practitioners in substance abuse translational research. The Institute engages new researchers in combining substance abuse research with evidence-based approaches to implementation. Such a training program has potentially substantial public health impacts given the sizable gap between the demand for implementation of high-quality evidence-based programs and the number of researchers focusing in this area [15, 16].

Glasgow et al. [17] called for new dissemination and implementation research-practice collaborations that include those community members and local practitioners who will be involved in implementation. This involvement is key to embedding health care advances into existing workflows and achieving the sustainability that often eludes implementation science efforts [17]. Accordingly, Glasgow et al. [17] expanded the definition of translational research to a five-stage process involving overlapping and interrelated stages: 1) T0 is the identification of a health/behavioral health issue; 2) T1 is the development of interventions for the health/behavioral heath issue; 3) T2 is the identification of the interventions' impact on the health/behavioral health issue; 4) T3 is the evaluation of the implementation of these interventions in practice; and 5) T4 is the evaluation of the effectiveness of the intervention across several applied settings. The Institute’s focus is primarily in T3 translational research, which includes “investigations designed to increase uptake and implementation of evidence-based recommendations into practice” ([17], p1276).

One of the most challenging aspects of implementing evidence into practice is that implementation stakeholders (i.e. providers, organizations) rarely have the skills to interpret and implement new research into practice [18]. In this paper, the authors describe part of the Institute’s success in terms of building capacity for implementing evidence-based practices by training current and future practitioners and researchers to understand translational research and implementation within community settings. We advocate that traditional measures of success focusing on academic productivity (grants and publications) fail to capture the breadth of impact of a translational research institute intending to train community practitioners, future health professionals, and researchers in translational research.

Institute overview

The central aim of the Institute is to develop innovative, applied research skills among behavioral health researchers and practitioners through a collaborative service-learning-focused approach. Institute scholars complete service-learning translational research and implementation science studies in collaboration with academic and community mentors and community behavioral health organization partners as part of the Graduate Certificate in Translational Research in Adolescent Behavioral Health Program. Implementation is not only relevant for academically based researchers but also has a pragmatic relevance for those who practice in the field. The intention of including both master’s and doctoral students in the Institute is to expose a broader range of professionals to the implications for translational research and implementation science in working agencies.

Service-learning studies are defined as research carried out as a service to the program. Scholars were expected to approach their projects based on what community mentors describe as the greatest need in their organization. The graduate certificate provided additional training for scholars, in most cases, outside of the requirements for their degree program. The program was designed as a graduate certificate to be available to enrolled graduate students and also available to community members seeking additional certification and/or knowledge for their current job. Additionally, making the program an official graduate certificate of the university created a sustainable program beyond grant funding.

Interested applicants applied to the Institute by submitting a personal statement, resume, two letters of recommendation, and official transcripts. Members of the Executive Committee then reviewed the applicants collectively and made final decisions regarding admission to the program. The Institute Executive Committee included the multiple principal investigators for the Institute and the project director who were responsible for the Institute’s design, implementation, and evaluation.

Two groups of mentors (academic and community) contributed to the delivery of the curriculum. The academic mentors provided support in research design and analysis, while the community mentors provided hands-on experience and real-world interpretations. Community agencies interested in participating in the Institute met with members of the Executive Committee to determine fit for the program, including the commitment to scholar project length and providing a community mentor at their site. Academic mentors were selected at the university based on areas of expertise that were applicable to the proposed research projects.

The graduate certificate program (15 credits) requires four continuous semesters, a total of 18 months, to complete. Scholars complete online coursework on adolescent behavioral health, translational research methods, and community-based participatory research, while also completing their in-person service-learning research project. During the first semester, scholars attend the Annual Research and Policy Conference on Child, Adolescent and Young Adult Behavioral Health where they participate in a networking event with community mentors and academic mentors. Scholars then rank preferences for projects, and the Executive Committee reviews scholar preferences and mentor expertise and places the scholars and mentors into their respective research teams. During the last semester of the program, graduating scholars also attend the conference (cited above) to present findings from their applied research studies. For additional information about the Institute, see the website, http://www.usf.edu/cbcs/cfs/itre/index.aspx [19, 20].

As part of the triadic mentoring relationship, scholars have the opportunity to gain experience in improving community readiness, uptake, and successful implementation of new programs and practices. Academic mentors provide experience in methodologies of data collection, research and evaluation methods, community consultation, and problem solving, while community partners provide the practical application possibilities and pitfalls of accomplishing such changes.

Measures for gauging the success of the program beyond academic productivity include real-world implementation and community outcomes. These measures are considered a step toward a more comprehensive understanding of the impact of translational research training on both scholars and community behavioral health organizations.

The purpose of this evaluation is to explore outcomes of the Institute beyond academic productivity in order to highlight areas of impact and suggest potential next steps in the field of translational research. The Translational Research Impact Scale (TRIS), developed by Dembe et al. [14], provides a framework for evaluating long-term and community-oriented outcomes of the Institute. The TRIS distinguishes three domains of potential impact: (1) research-related impacts, (2) translational impacts, and (3) societal impacts. Expansion to translational and societal impacts lends a broader perspective to evaluation, moving beyond academic productivity to consider implementation and impact as key outcomes of translational research training. For a full list of potential impact indicators in each sub-domain of the TRIS, refer to Dembe et al. [14].

Methods

Participants and data collection

Scholars

The Institute engages in ongoing process and outcomes evaluation in its efforts to inform practitioners and future researchers about the field of translational research and implementation science. The current study reports evaluation measures completed to date with the first two cohorts of scholars enrolled in the graduate certificate program. Evaluation is mainly focused on cohort #1 scholars, for whom information about career trajectory and longer term impact has already been collected. Cohort #1 included a Masters of Social Work student, six Masters of Public Health students, four employees of community agencies, two doctoral students in public health, and one doctoral student in criminology. Insights from cohort #2 scholars (similar in background), as well as academic and community mentors, are also included.

Cohort #1 scholars (completed Institute participation in May 2014) were interviewed 1 year after completion of the Institute graduate certificate program. Questions focused on long-term outcomes of participation in the Institute, including current career, current interests and engagement in translational research, dissemination and research activities, and other themes of interest. Cohort #2 scholars (completed Institute participation in May 2015) were interviewed about their experiences in the Institute and plans for future research, career aspirations, and beliefs about how the Institute influenced their careers and professional aspirations overall.

Community mentors

For each cohort, Institute scholars were matched with a community partner organization focused on behavioral health in order to complete the translational research and implementation science study. For cohort #1, five community behavioral health organizations agreed to host a team of scholars and provide an on-site community mentor to work directly with scholars on the conceptualization and implementation of the translational research study. The on-site community mentor provided oversight on the nature and scope of the project, while also assisting the scholars with ongoing data collection and analysis activities. For cohort #2, all but one of the original five agencies continued their involvement with the Institute.

Academic mentors

In addition to being paired with a community mentor, scholars were also paired with academic mentors, forming a triadic mentoring relationship. Academic mentors were tasked with providing consultation and oversight of research methodology including qualitative and quantitative approaches for data collection and analysis. In particular, academic mentors provided oversight on methodology in order to ensure the service-learning translational research studies best matched the needs of the behavioral health organizations. Academic mentors were paired with teams of scholars whose proposed research studies matched at least one of their areas of research expertise.

Data analysis

The purpose of this evaluation was to assess longer term impacts of the Institute on scholars as well as from the perspectives of academic mentors and community partners. This evaluation was approved by the University of South Florida Institutional Review Board and participants provided verbal consent to participate. Semi-structured qualitative interviews were audio recorded and transcribed verbatim. A member of the Executive Committee trained the interviewers. Interviewers conducted practice interviews with Institute staff and met to discuss issues that arose in order to align strategies prior to conducting the interviews.

Sample size was limited by the total number of people participating in the Institute program. In contrast to quantitative sampling in which large numbers of participants are required to achieve significance, relatively small numbers, as few as twelve in some instances, of participants are typically needed in qualitative research to achieve what is referred to as “thematic saturation” [21,22,23,24]. Thematic saturation is achieved when additional interviews no longer uncover unique themes in participants’ comments. Because of the narrow scope of the questions explored in this evaluation, we believe that the total number of interviews (n = 33) was sufficient for the purposes of this evaluation.

All interview transcripts were coded manually using both inductive and deductive approaches [23,24,25,26] for themes related to outcomes and impacts of interest for the Institute evaluation. Pre-determined areas of interest used to identify themes included: (1) career aspirations, (2) current career, (3) dissemination activities, (4) education activities/aspirations, (5) grant-writing activities/aspirations, and (6) influence of the Institute on Scholars’ lives overall. The importance of networks gained through Institute participation was added as an emergent theme of central interest to scholars and community mentors. Two coders met multiple times to discuss coding and establish reliability. This reliability procedure was intended to critically reflect on disagreements and improve analytical consistency, not to measure percentage of agreement [27, 28]. Qualitative analysis of codes was performed by an independent analyst not affiliated with the Institute with expertise in qualitative analysis methods and theoretical analysis [29,30,31]. Involving a third party in analysis helped to reduce bias and bring a fresh perspective to assess Institute outcomes.

After initial coding and analysis, the Translational Research Impact Scale (TRIS) [14] was used to frame the potential impact of translational research efforts of the Institute. This evaluation demonstrates the Institute’s impact in all three domains of the TRIS evaluation framework (see Dembe et al. [14]).

Results

Institute scholars were recruited from health- and social-services-related graduate academic disciplines, programs, and fields, such as criminal justice, education, nursing, psychology, public health, behavioral health, rehabilitation and mental health counseling, and social work. Of the fifteen scholars in cohort #1, twelve agreed to be interviewed for this evaluation. Of the twelve scholars in cohort #2, seven agreed to be interviewed for this evaluation.

Community mentors were full-time employees at local or regional child and adolescent behavioral health service agencies. We use behavioral health service agencies to refer to organizations that provided mental health, substance use, and/or educational programs, in this case specific to children and adolescents. Three of the agencies provided direct behavioral health services. One agency disseminates and evaluates an evidence-based curriculum focused on preventing substance abuse. The final agency was the social work services division of a local county school system. All five community mentors representing participating community behavioral health service agencies participated in the 2014 evaluation. Three out of the four community mentors participated in the 2015 evaluation.

Of the academic mentors, two were psychologists with backgrounds working with community and school-based mental health intervention programs. One academic mentor had a background in mental health policy and social/health psychology and extensive experience working with community behavioral health agencies on evaluation projects. Another academic mentor was a sociologist with expertise in behavioral health statistics. All six of the academic mentors involved with the cohort #1 scholars participated in interviews during the 2014 evaluation. Three of the four academic mentors involved with the cohort #2 scholars participated in interviews during the 2015 evaluation.

Using the domains of the TRIS framework [14] (research-related impacts, translational impacts, and societal impacts) as a guide, the authors describe the range of ways participation in the Institute impacted both scholars and community mentors. Overall, scholars and community mentors reported increased interest in translational research, implementation science, and substance abuse research as a result of participating in a research project as part of the Institute service-learning component, as well as development of valuable professional networks and ongoing collaborative partnerships. The impacts in each of the TRIS domains follow and quotes from scholars, community mentors, and academic mentors illustrate their respective impact in specific areas of the TRIS domains.

Research-related impacts

Domain 1, research related impacts, is the most extensive domain in the TRIS. Research-related impacts include areas such as (1) identification of research needs, (2) development of innovative methods, (3) improvement of research conduct and/or management strategies, (4) improvements in analysis and synthesis of results, (5) increases in the number of grant submissions and publications among translational researchers, (6) new, marketable discoveries, and (7) successes in research dissemination to appropriate audiences. Table 1 contains illustrative quotes from participants to show research-related impacts resulting from Institute-applied research projects.

Table 1 Illustrative quotations to demonstrate areas of research-related impacts
Full size table

A range of new research networks and collaborations were formed as a result of collaborative mentoring and community-based participatory research studies conducted as part of the Institute graduate certificate program. Community mentors, representing five different community behavioral health organizations collaborated with Institute scholars on community-based applied research studies. Although some individuals had prior relationships with the Institute’s faculty, community mentors and academic mentors reported improved collaborative relationships as a result of the Institute service-learning component [32].

The triangular research mentoring relationship, including scholars, academic mentors, and community mentors, reportedly influenced scholars’ success and helped them develop and refine competencies in public health and translational research [19]. One year after participation in the Institute, scholars reported that networks gained through experience with the Institute were instrumental in obtaining their present positions and both scholars and community mentors reported those networks strengthened their commitment to pursuing community-engaged work initiatives.

To date, four cohorts of scholars have created and implemented applied research studies in collaboration with community mentors. The service-learning studies that were completed have positively impacted the operations of participating community organizations [20]. The translational research studies completed by Institute scholars in collaboration with community mentors ranged from exploring the integration of behavioral health and primary care services (traditionally delivered separately in the United States) to identifying factors and mechanisms for adapting evidence-based practices in schools. Scholars and community mentors reported heightened awareness of the need for practical application of evidence-based research.

In the context of translational research, individual mentoring can be difficult and thus not conducive to collaborative training. The Institute developed a collaborative mentoring approach to address this challenge and provided broader training to scholars as well as offering beneficial service-learning research to community behavioral health organizations [19]. Emphasis on the “real-world” or actual community agency experience provided by the Institute was a key theme in scholars’ descriptions of the overall impact of the Institute on their lives and career aspirations. Scholars reported that as a result of participation in the Institute, they had a better appreciation for the importance of research in applied community settings, as well as the need for evidence-based understanding prior to implementation and intervention.

Inclusion of community mentors and practice-oriented scholars led to widespread diffusion of understanding of the techniques involved in conducting and implementing translational research. Scholars reported giving presentations to local policymakers and community organizations. Those scholars working for health organizations also reported bringing techniques learned from participating in the Institute into their work and educating their colleagues on the value of evaluation.

Five scholars from cohort #1 reported presenting study results at national conferences. Several manuscripts are in preparation, authored by scholars and mentors from cohorts #1 and #2. Thus far, one manuscript has been published by an Institute scholar team [33]. Scholars cited the short timeline of participation in the Institute, the need for more incentives to publish, and inexperience in academic writing as barriers to manuscript preparation, submission, and subsequent publication.

One scholar noted that over-emphasis on academic productivity can frustrate practice-oriented scholars and de-emphasize outcomes that are of importance to community organizations such as programmatic development or sharing results with other providers.

Sometimes we get really wrapped up in formal research, in publishing and getting our names on things. That kind of left a bitter taste about that, not specifically from the Institute just like in general through my master’s program. I think maybe that’s why I’ve been not as involved or like eager to go to conferences or eager to be in the American Public Health Association or anything, just because I feel like we end up getting kind of really lost in that. I’m much more of like a one-on-one kind of person, like hands-on on the ground type work. (Cohort #1 Scholar)

To balance the different outcomes valued by academia versus community organizations, one key outcome for all projects included in the Institute was a presentation at a national conference held in Tampa, FL (Annual Research and Policy Conference on Child, Adolescent, and Young Adult Behavioral Health). Disseminating project results at this conference created an opportunity to speak to both academic- and community-based audiences.

Translational impacts

Domain 2, translational impacts, includes potential impact in areas such as (1) incorporation of discoveries from bench science into human or animal studies, (2) incorporation of clinical trial results into clinical or practical guidelines, (3) recruitment and preparation of new translational researchers, and (4) improvement in evidence-based health care service and delivery, patient outcomes, and positive health behaviors, as a result of research translation. Table 2 provides illustrative quotes from Institute participants regarding translational impacts.

Table 2 Illustrative quotations to demonstrate translational impacts
Full size table

The Institute’s inclusion of graduate students and working professionals from a range of health- and social-services-focused disciplines resulted in a diverse cohort of professionals trained in translational research methodology not only from the university but also from community agencies. Whereas transdisciplinary research across university departments is important in furthering research and translation, members of community-based organizations involved in implementation also benefit from training in translational research and implementation science methods. As earlier Institute-focused publications describe, the processual development of the Institute, including a community agency network and a translational research training program, have contributed toward the goal of increasing the number of people working in health care who have been trained to understand the importance of evidence translation and research [19, 20, 32].

Societal impacts

Domain 3, social impacts, includes potential impact areas such as (1) strengthening and refining of health-related policies and procedures, (2) improvements in community health, empowerment, and economic conditions to reduce health disparities, and (3) development of new community-based participatory programs and partnerships geared toward effective and meaningful implementation. For illustrative quotes related to societal impacts resulting from Institute projects, see Table 3.

Table 3 Illustrative quotations to demonstrate impacts related to local community and society
Full size table

Service-learning applied research studies completed as part of the Institute’s graduate certificate and training program were collaboratively designed and implemented by scholars and community mentors and academic mentors. The emphasis on mutual benefit was designed to ensure the sustainability of community-based behavioral health programs and partnerships. While most scholars did not continue to collaborate with the community mentors after graduating from the program, the community mentors continued partnerships with the university through ongoing service-learning projects with new cohorts of Institute scholars. Rather than attempts to create new behavioral health programs, the Institute’s training program was designed to train new researchers as well as make meaningful contributions to the function and applicability of existing behavioral health programs. Scholars have presented study outcomes and insights to local policymakers and community behavioral health organizations. Such presentations contribute to participatory decision-making and policy development. Community mentors described impacts to their organizations in terms of understanding the value of applied research and collaboration with the University, as well as of the need for adaptation and innovation of their ongoing operations.

Examples of outcomes from service-learning translational research projects completed as part of the Institute’s graduate certificate program include the following: (1) development of a training for providers focused on appropriate mechanisms to adapt a school-based evidence-based practice curriculum; (2) implementation of a professional development training program focused on mental health counseling; and (3) identifying facilitators and barriers to integrating primary care and behavioral health services.

The Institute’s emphasis on community-university partnerships with a community-based participatory research foundation was designed to broadly impact child and adolescent behavioral health issues [32]. Community behavioral health organizations involved in the Institute reported benefits as a result of collaboration with scholars and improved methods for serving their targeted at-risk populations.

At the time of the evaluation, cohort #1 scholars were employed in a range of behavioral health organizations and university departments. Current careers included the following: technical assistants on health focused grant projects; reporting and analytics for a health care organization; senior qualitative analyst for a health care organization; administrator for a substance-abuse-focused organization; curriculum development and implementation specialist for a substance abuse outreach organization; manager for a children’s special needs organization; and research fellow for a federal healthcare organization.

As noted above, one of the most challenging aspects of implementing evidence into practice is that implementation stakeholders (i.e., providers, organizations) rarely have the skills to interpret and implement new research into practice [18]. Part of the Institute’s societal impact is in providing training, particularly in T3, to people employed in various community organizations.

Discussion

Implementing a multi-component, applied service-learning research experience, in addition to education or perhaps curriculum-based translational research, holds the potential to expand the public health and behavioral health workforce in needed areas [34, 35]. To monitor the success of such programs, however, knowledge about translational research and implementation science must be considered along with evidence of productivity [36]. Evaluation of translational research education training must therefore include measures of success beyond conventional academic metrics of peer-reviewed publications. In this paper, several areas of impact beyond academic productivity are highlighted. Dimensions such as community engagement, network building, and practical application of evidence are areas of potential impact in translational research and implementation science; however, such effects have not been emphasized or measured in translational research education training [13].

Fortunately, there are resources available to encourage translation of research into practice. The National Collaborating Centre for Methods and Tools [37] has developed a series of resources for practitioners, including education and self-assessments, to encourage their uptake and use of evidence-based practices. Approaches to evaluating the success of translational research education training require sufficient flexibility to accommodate individuals and individual institutions, while still retaining enough rigor to demonstrate that a program is meeting established objectives and competencies [2]. Another potential outcome of translational research education could be utilizing evidence to inform policy making. The SUpporting POlicy Relevant Reviews and Trials (SUPPORT) Project provides a series of resources targeting policy-makers to encourage their use of evidence-informed decision-making in policy development [38].

Training and mentoring new researchers in implementation science and in translational research are increasingly emphasized in health equity research, as evidenced by the many current programs in existence for training both students and faculty members [1, 6, 8, 10, 39,40,41]. To facilitate innovation in translational research in clinical and services research, researchers from various disciplines must be willing to engage in collaboration beyond the defined margins of their discipline [9]. Although most education training is focused on early career research faculty, Wooten et al. [4] suggest that engaging graduate-level scholars in translational research may be the key to inspiring a new cohort of investigators in this field. A critical hurdle in translating knowledge to practice is the gap that exists between academic publication goals and the policy-oriented interests of public health and behavioral health directors and service agency administrators. There is a need to establish effective iterative loops in translation between researchers and those involved in implementation in order to address shared goals [15].

Limitations

This paper is limited by the self-reported nature of the qualitative data collected. Qualitative investigation is well-suited, however, to providing in-depth information about participants’ experiences. For the purpose of this evaluation, qualitative interviews were intended to elicit information about stakeholder experiences in the Institute. Although this paper does not provide a comprehensive evaluation of the Institute, these findings about Institute scholar experiences and the need for alternative measures of success will likely be of interest to similar current and future educational training programs in translational research [8, 39,40,41]. Findings related to the relevance of translational research training to members of community behavioral health organizations also point to the importance of training in these skills beyond academia.

Conclusions

Over the course of this evaluation, the authors identified several drawbacks in emphasizing academic measures of productivity rather than community outcomes in evaluating translational research training programs. Common measures, such as numbers of grant applications and awards, are less revealing of success when applied to practice-oriented graduate and professional degree programs and community organizations. In this paper, the use of the TRIS domains to identify areas of potential impact suggests ways of broadening measures of success in the context of a translational research educational training program designed for graduate students, community partners, and academic mentors. This triadic, applied approach to translational research training has potential to broaden the reach of translational research in several ways. After participating in training programs, researchers are better prepared to work in interdisciplinary environments and to successfully implement evidence-based programs in applied community practice settings. Further, training not only graduate students but also members of stakeholder organizations can facilitate improved communication and understanding and lead to stronger community/university partnerships.