Skip to main content
Download PDF

From Planning to Implementation of the YouthCan IMPACT Project: a Formative Evaluation

The Journal of Behavioral Health Services & Research volume 47, pages 216–229 (2020)Cite this article

Abstract

In order to improve the youth mental health system, there is an international movement toward developing community-based service hubs that provide integrated, collaborative care to youth. However, the implementation of multisystem collaboration is complex and can be hampered by barriers. This paper presents a formative evaluation of the YouthCan IMPACT integrated youth services project based on the Consolidated Framework for Implementation Research (CFIR), to identify facilitators and barriers to successful implementation. Results highlight that previous positive working relationships along with collaborative investment of resources from partnering organizations are essential to implement an integrated youth service model. In addition, it is important that representative members of all key stakeholder groups, including staff, youth, and caregivers, be involved in the development and execution of the project to ensure effective implementation. Attention to the facilitators and barriers to implementation may help teams seeking to implement highly collaborative, integrated models of service delivery for youth in the community.

Introduction

Adolescence marks the developmental transition from the dependency of childhood to the independence of adulthood.1 During this critical time, an estimated 12.6% of Canadian youth experience mental illness.2 Mental illnesses can be chronic and disabling, with wide-reaching effects on quality of life, productivity, and functioning.3 Adolescence is an especially important period, since over 70% of mental illnesses first present before adulthood.4 Furthermore, suicide is the second leading cause of youth mortality.5 Despite the clear need for prevention and early intervention services, youth worldwide face barriers with regard to service acceptability, accessibility, availability, and equity.6

In Ontario, Canada, the youth mental health system has been described by youth as lacking in resources and ineffective, with disjointed services.7 In addition, many Canadian families seeking services from the child and youth mental health system have found it difficult to identify and navigate available and appropriate services, and they experience long waitlists, as well as a deficiency of access to coordinated care.8 Service users’ (e.g., youth and caregivers) involvement in service design and evaluation has been shown to result in better experiences and outcomes, yet their voices are rarely integrated.9 Thus, a crisis of care has emerged, which requires innovative solutions.

In order to improve the youth mental health system, diverse agencies are increasingly collaborating to provide youth with a wide range of services based on their needs. Throughout the world, service problems in youth mental health care are being addressed through integrated youth service hubs.10,11,12,13 These integrated service hubs bring together the expertise of diverse service providers in a “one-stop shop” format, with a focus on increasing access and reducing the siloing of services. These models typically include services for mental health and substance use issues, primary health care, and social and vocational services.11, 12 Examples of integrated youth service hub models in Canada include YouthCan IMPACT in Toronto,12 Foundry in British Columbia,14 ACCESS Open Minds, a pan-Canadian initiative,15 and Ontario’s Youth Wellness Hubs Ontario initiative.16 Examples in other countries include headspace in Australia,17 Jigsaw in Ireland,18 and the Youth One Stop Shops in New Zealand.19 California and Quebec both recently announced plans to develop such models.20, 21

The YouthCan IMPACT initiative in Toronto, Canada, includes an integrated youth services model with an embedded pragmatic randomized control trial (pRCT).12 The pRCT compares the YouthCan IMPACT integrated youth services (YCI-IYS) model (Fig. 1) to out-patient hospital-based psychiatric services, to determine the impact on clinical and functional outcomes, user satisfaction, and cost-effectiveness. The YCI-IYS model is community-based and was co-created with youth,22 family members, and other stakeholders. It aims to provide accessible, youth-friendly,23 evidence-informed mental health services, tailored to the needs of individual youth and offered in the context of other health and social services.

Figure 1
figure 1

YouthCan IMPACT service delivery model

Full size image

The YouthCan IMPACT integrated youth services model12 is unique in its design and co-creation with various stakeholders to establish youth- and family-friendly services that are relevant to youth and better able to meet their needs. Based on feedback from stakeholders, the stepped-care model includes a brief standardized needs assessment, solution-focused brief therapy (SFBT),24, 25 dialectic behavioral therapy (DBT) skills groups,26 psychiatric services, and other diverse services available on site, such as nurse practitioner services, care navigation, individual and group peer support, e-resources, and interventions for caregivers, including the Family Connections DBT-based caregiver group.27 The YCI-IYS model is consistent with the movement toward integrated youth service hubs; however, it is unique in that it was co-created by local service providers, youth, caregivers, and researchers with a view to creating an accessible, youth- and family-friendly suite of services that leverages available services to be locally feasible, while also being highly relevant to youth and caregivers, and therefore better able to meet their needs.

Achieving the expected outcomes from the implementation of health service interventions can be significantly hampered by challenges faced in real-world settings.28, 29 Thus, conducting a formative evaluation during early project phases can be critical to assessing the potential for the future real-world success of a novel service model. A major benefit of a formative evaluation is that it does not examine traditional summative endpoint measures; rather, it pinpoints potential or actual facilitators and barriers to implementation efforts.30 Thus, a formative evaluation can provide important information for achieving replicability and successful implementation of novel service models, like the YCI-IYS model.

The Consolidated Framework for Implementation Research (CFIR)28, which has previously been used to guide formative evaluations31, 32, proposes five main domains relevant to implementation success—intervention characteristics, outer settings, inner settings, characteristics of individual, and process—each further divided into constructs influencing implementation.28 The CFIR has been effectively used to develop interview questions and frame data collection and data analysis for implementation evaluations.33 Furthermore, it has also been used to guide and examine initiatives in community-based youth mental health.34 Accordingly, the CFIR framework was used in this study to conduct a formative evaluation of the YouthCan IMPACT initiative, including development and implementation of the YCI-IYS model and the related pRCT.

Objectives

The main objectives in this study were (1) to describe and examine the YouthCan IMPACT initiative startup process and (2) to determine the facilitators and barriers to implementation and ongoing operations.

Consistent with implementation science recommendations and the call for formative assessments to inform implementation,30 it was expected that this evaluation would enhance the implementation of the YouthCan IMPACT initiative and produce knowledge about the implementation process that could be used to optimize the potential launch of additional local YCI-IYS sites. Furthermore, the knowledge generated could be used to assist other groups in implementing similar hub-based models or other youth mental health interventions.

Methods

Interviews

Data were collected through semi-structured interviews. Twenty-six members of the YouthCan IMPACT team were interviewed for this evaluation. Participants had diverse project roles, spanning from scientists and partnering agency leads to youth and family representatives and direct clinical service staff. Ethical approval to conduct the study was obtained from the Centre for Addiction and Mental Health Research Ethics Board. All participants were informed about the process and written consent was obtained. Interviews were conducted by research staff at a time and place convenient for the participants. Where it was not possible to conduct interviews face-to-face, they were conducted via telephone. Questions for the interviews were developed based on (a) CFIR domains and (b) information collected from a review of protocol documents, archived meeting minutes, and committee meetings/site visits. Question sets were adapted to the role of each interviewee. The information gathered from the interviews was analyzed and categorized based on the CFIR domains and constructs.28

Surveys

In addition to semi-structured interviews collected from YouthCan IMPACT team members, 26 direct service staff and managers at the clinical sites implementing the YCI-IYS model were surveyed at the beginning of implementation. Ethical approval for the surveys was also obtained from the Centre for Addiction and Mental Health Research Ethics Board. The participants were approached in person, consent forms were administered, and participants completed a survey. The survey questions included demographic information and the Service Provider Adopter and Innovation Characteristics Questionnaire (SPAICQ).35, 36 The SPAICQ was created by adapting previously used measures35,36,37,38,39 and formulating new items based on the theoretical constructs from the literature. The SPAICQ has three subscales examining adopter characteristics (concern, self-efficacy, and attitudes) with a total of 21 items, as well as three subscales examining innovation characteristics (complexity, compatibility, and relative advantage), with a total of 14 items. Response options are on a 7 point Likert scale ranging from 1 “strongly disagree” to 7 “strongly agree.” Table 1 provides sample items in each subscale. Demonstrating reliability, the internal consistency coefficients for each subscale (Cronbach's alpha) were found to range from 0.76 to 0.87 (Table 1). The data obtained from the survey were analyzed using SPSS 24.

Table 1 Service Provider Adopter and Innovation Characteristics Questionnaire
Full size table

Results

Results describe the qualitative findings, with quantitative findings embedded to facilitate triangulation. Complete quantitative findings are summarized in Table 1.

Description of the startup process

A number of researchers, service providers, clinicians, youth, and family members came together through their strong interest in youth mental health, their complementary expertise, and prior working relationships to develop an integrated youth services project in response to a call for grant proposals. Two youth with lived experience participated as co-investigators and co-creators of the initiative and a broader range of youth were engaged in project development through a pre-existing National Youth Action Council (NYAC).40 Similarly, one caregiver was engaged as a co-investigator on the project and a Family Advisory Group was established to provide a broader range of caregiver feedback. Service partners included child and youth mental health agency representatives, youth social service providers, primary care providers, and hospital-based youth psychiatry clinicians. Other stakeholders who could inform the methodology, design, economic analysis, and policy context were also engaged for the project. After funding was awarded by the Ontario SPOR Support Unit, research staff and additional youth were hired as YouthCan IMPACT staff to support implementation of the project and a local Youth Advisory Group was established. In addition, sub-project working groups were developed to ensure the necessary leadership and accountabilities were in place to achieve the project deliverables. To ensure effective collaboration and communication between team members, a governance model was created (Fig. 2).

Figure 2
figure 2

YouthCan IMPACT governance model

Full size image

In order to develop the YCI-IYS model and select the interventions that would be offered as part of the model, community partners, youth, and caregivers worked together with the researchers to develop a pathway of services incorporating sustainable interventions already working effectively at some or all of the community agencies. These interventions were then organized into a comprehensive stepped-care YCI-IYS service delivery model (Fig. 1). In addition, the roles and responsibilities of partnering agencies were outlined, and memorandums of understanding, contracts, and budgets were established. As well, an intervention manual that was created for training on-site staff about the model and fidelity measures were put in place to support consistent implementation of the model. An incremental rolling start approach was taken, where agencies possessing the capacity started implementing the model while others continued to develop and become ready to implement. Once workflow processes were established, services were promoted in the community and expanded. Fidelity check points were also implemented to permit ongoing implementation monitoring and adjustment.

Barriers and facilitators of implementation

To develop an in-depth understanding of the barriers and facilitators of implementation of the YouthCan IMPACT initiative, we used the CFIR framework.28 The barriers and facilitators were identified via both interviews and questionnaires, and are presented based on the five domains within the framework provided by CFIR.28

Domain 1: intervention characteristics

According to interviewees, the YCI-IYS model was primarily designed by partnering agencies, youth, and caregivers with support from the project leads and project coordinator. Interventions proposed by youth, caregivers, and community agencies were included after ensuring evidence-based support. Thus, it was perceived as a “bottom-up” community-based model rather than a “top-down” imposed research initiative. This was identified as an important aspect of the model for community service-provider uptake, and service providers who were not involved in model creation were informed of this aspect of the development process to facilitate uptake.

Key constructs of an intervention considered as part of CFIR include the strength of evidence supporting the intervention and its perceived relative advantage over interventions in use.28 In this project, interviewees reported that partner buy-in to the model was enhanced by evidence of international success of similar hub-like models,41, 42 along with the evidence base for the individual interventions included in the pathway.25,26,27, 43 In addition, partnering organizations had a history of using these interventions successfully. Many team members reported that they had been frustrated with the youth mental health system at the time and that they were highly motivated to provide better services for youth. These findings are reflected in the quantitative data; surveys of hub staff and managers revealed that service providers perceived the YCI-IYS model to have high relative advantage over existing youth mental health services (relative advantage subscale mean score = 6.15, S.D. = 0.83; see Table 1). Thus, qualitative and quantitative data converge to suggest that the implementation of the model was facilitated by strong evidence-based support and a high perceived relative advantage over the existing youth mental health system.

Adaptability of the model, which is another key CFIR feature, was identified as a strength by interviewees, who indicated that the model was broken down into core components (SFBT, DBT, and psychiatry) and adaptable components (other services available at each site). They also reported, however, that a challenge that emerged during the initial implementation phase was balancing adaptability with fidelity. In order to facilitate fidelity, the team aimed to partner with organizations that already provided the core services. When an agency was not able to provide the core services, other partnering organizations leveraged their existing service capacity to provide the missing services for these agencies or training in these services. By leveraging the strengths of each partner agency in the core components, interviewees reported it was possible to enhance fidelity while maintaining adaptability. Overall, according to interviewees, the adaptability of the model was considered a facilitator of these early phases, as it ultimately allowed for the implementation of the model across three different sites with variable local demands, different combinations of services provided, and different existing capacities (e.g., financial resources, staff size, and existing interventions).

The complexity of an intervention can be a barrier to its effective implementation.28 Qualitatively, interviewees identified that the YCI-IYS model is streamlined, with only one step to access walk-in services and a further step to access needs-based services. Thus, the model itself is not complex. The most complex aspect of the YCI-IYS model, as expressed by the team members interviewed, is the collaboration among multiple organizations with different intervention processes at each organization. This was also reflected in the quantitative data, where the service providers felt the model was neither highly complex nor highly straightforward to implement (complexity subscale mean score = 4.14, S.D. = 1.16; Table 1). Interviewees qualified this finding by indicating that the relative complexity of the intervention was reduced by leveraging the relationships with multiple organizations, which helped to provide inexperienced staff with experience by allowing them to “shadow” experienced staff in other organizations.

Another challenge identified as present during the implementation phase was the cost of the intervention, since the services that could be feasibly offered were limited due to the budgetary restrictions. Again, leveraging existing services helped address this barrier. Despite this challenge, interviewees indicated that staff still highly regarded the design of the model and felt that it was not compromised by the budget.

Domain 2: outer settings

Interviews revealed that the YouthCan IMPACT project was created during a time of movement toward community-based mental health services for youth in Ontario and across Canada, reflecting the CFIR domain “outer setting.” Strategic planning by the provincial government at the time44 aimed to ensure children, youth, and families had accessible responsive local community-based mental health services. Lead agencies responsible for the provision of core youth mental health services in their communities had recently been identified. Participants felt that the YCI-IYS model, which was also aiming to strengthen community-based mental health services to make them more accessible for youth, was well aligned with this movement. In addition to the provincial government plan,44 there was a national movement to conduct “patient-oriented” research (Strategy for Patient-Oriented Research (SPOR)45) with both provincial and federal funding streams as well as cross-jurisdictional interest in health systems integration. Interview participants highlighted that YouthCan IMPACT goals aligned with these aspects of the external environment, facilitating partner participation and uptake of the model.

According to interviewees, development and implementation of YouthCan IMPACT was facilitated by the high degree of social capital and strong ties with local community mental health organizations that the project leads had previously established. They also had ties within their affiliated hospitals and multiple domains of academia. Furthermore, members of the YouthCan IMPACT project had boundary-spanning roles and broader experience due to engagement with multiple organizations. Since increased social capital and relationships with individuals who possess boundary-spanning roles have been known to increase the efficiency of implementation,46, 47 the current project appeared to have the benefit of robust outer settings facilitators.

Domain 3: inner settings

The way in which an organization is structured can greatly impact the implementation success of that organization.48 In order to facilitate the necessary formation of a cohesive group to implement a project as highly collaborative as the YouthCan IMPACT project, interviewees highlighted the importance of the governance model, with a core team of the five project leads together with other team members, as well as the subdivided, specialized working groups (Fig. 2), in providing structure to the inner setting. Each working group included a project lead and the project coordinator. According to interviewees, this helped to create decentralized decision-making opportunities, as well as to ensure effective communication across the working groups and to the core team. In addition, interviewees indicated that the project co-ordinator served as a champion for the project outside of the governance structure,49, 50 facilitating and communicating effectively across the partner organizations.

Relationships play an important role in implementation51 and this notion was strongly supported by the YouthCan IMPACT team. Interviewees reported that positive previous working relationships were a key factor in the determination and persistence of the partnering organizations. They indicated that this aspect of the inner setting was a key facilitator of startup and implementation, as they felt they could trust their teammates, voice their ideas, and disagree openly, while they felt confident that the sharing of resources would result in mutual benefit. Beyond the relationships among project leads and community partners, positive working relationships were found to be critical to meaningfully engage youth and families. There is a power differential between service users and their health services providers, making service users vulnerable.52 Since the project leads were experienced in maintaining effective relationships with youth and caregivers,40 youth and adult partners alike felt the respectful and open atmosphere during the decision-making process in this project allowed all parties to contribute equally and meaningfully.

Interviews also revealed that leadership engagement in the YouthCan IMPACT project was a key facilitator and one of the main drivers of the startup and implementation processes. Key leaders within partnering organizations, who had decision-making power on behalf of the organizations, were engaged, which allowed the development process to move efficiently. Through surveys of hub staff and managers, quantitative findings revealed that the YCI-IYS model was felt to be highly compatible with the work of their organizations (compatibility subscale mean score = 6.03, S.D. = 0.85). Interviewees indicated that the project was given a high degree of priority from organizational leaders and staff; although the project required frequent meetings and a substantial time commitment, resulting in possible short-term reductions in productivity, organizational leaders had the foresight to provide this time on an in-kind basis for the resulting long-term gain. This endurance, coined “managerial patience,” has been found to result in implementation success.53

One of the challenges reported by interviewees in the YouthCan IMPACT project was a lack of control over the architecture of the partnering organizations. Research has shown that stable teams are more likely to implement a project successfully.51 This was observed in the current project, where organizations with consistent management throughout the startup phase reported finding it easier to implement the project than partners with fluctuating management resources; organizational mergers occurring during the startup phase posed challenges in terms of structural stability. In addition, interviewees indicated that communication within partnering organizations, i.e., between higher level management and direct service staff, was challenged by decisional changes over time during the initial project phases. It was found to be helpful to include direct service staff in the decision-making process and/or provide finalized information to direct service staff in order to facilitate implementation. Interviewees observed that there was a shift in the YCI-IYS model implementation sub-project working group as it moved from upper-management community representatives during planning and design to operational, clinical managers during the implementation phase.

Domain 4: characteristics of individuals

Individuals must have sufficient information about an intervention to be willing to adopt it.46 While the members of the YouthCan IMPACT core team and working groups involved in the design of the model had “expert” knowledge of the model, the direct service staff, despite the startup training provided, expressed that they would have benefitted from more information about the project during the startup phase. These same staff, however also held attitudes typically found to facilitate adoption and implementation. For example, the quantitative data reveal that respondents perceived taking an integrated approach to mental health care to be important (concern subscale mean score = 6.57, S.D. = 0.71) and held positive attitudes about integrated collaborative care (attitudes subscale mean score = 5.49, S.D. = 0.86).

Self-efficacy positively affects individual’s willingness to adopt an intervention, their persistence in face of difficulty, and their performance.54 In the current project, quantitative data showed moderately high self-efficacy among staff to deliver the selected interventions (self-efficacy subscale mean score = 5.16, S.D. = 1.47). In order to further improve self-efficacy, staff were provided time and experience by shadowing other experienced staff. In addition, qualitative data revealed that that organizational managers realized the importance of hiring candidates experienced in providing services included in the project. For example, partner organizations specifically hired staff with experience offering brief walk-in services, a pillar of the YCI-IYS model. Interviewees reported that individuals with prior experience with these services expressed greater confidence in their ability to offer the YouthCan IMPACT services.

One important facilitator during early phases identified by interviewees was the individual traits of team members involved in the YouthCan IMPACT project. Interviewees indicated that during the development phase, the individual team members had a “yes” mentality, i.e., when encountering issues during implementation, rather than starting the discussion with “no,” they tried to develop solutions based on available resources. For example, if organizations felt unable to support a core component of the model, the team members offered creative solutions to provide the needed support. Interviewees highlighted the strength of individual characteristics as facilitating implementation of the model, as the individuals formed a highly determined group with a desire to be successful and persistence to accomplish the team’s goals.

Domain 5: process

According to the interviewees, during the planning process of the YouthCan IMPACT project, the main focus was on enabling and empowering community partners for implementation. Some examples provided include the team’s efforts to ensure that the community partners guided the selection of interventions in the YCI-IYS model, preparing staff members to implement the model by providing them with training and conducting an incremental implementation, allowing time to build the capacity of the organizations to provide the services included in the project. Interviewees reported that these processes were key: the quality and extent of planning for implementation during the startup phase facilitated the implementation through clear and feasible design, staff preparation, and incremental execution.

Some interviewees noted that a facilitator to building this group of team members was the project leads’ experience and dedication to engaging youth and caregivers in mental health research and program development.40, 55 This helped the rest of the YouthCan IMPACT team to support the youth and caregiver’s ongoing involvement since they were guided by experienced leadership. Interviewees reported that another major facilitator to youth and caregiver engagement was the work done by the youth and caregiver co-creators themselves to engage and maintain a large pool of involved individuals in the Youth Advisory and the Family Advisory Groups, respectively.

A key construct of the implementation process is reflecting on and evaluating the progress of the implementation.28 Interviewees highlighted the fact that the YouthCan IMPACT team conducted regular debriefing meetings with clinical hub managers and the project coordinator, which allowed for the identification and tackling of implementation challenges. Such challenges included clarifying the needs-based care model and updating the implementation plans that evolved during the startup process to ensure fidelity of all core components of the YCI-IYS model. Interviewees indicated that with efficient communication, the fidelity of the finalized model among direct service providers was supported, which led to uptake of the final model.

Discussion

The main objectives of this study were to describe the startup process of the YouthCan IMPACT project and examine the barriers and facilitators encountered during this process. Using surveys and interviews, information was gathered and analyzed based on Damschroder’s CFIR constructs for implementation research.28 The results provide specific findings applicable to the ongoing optimization of the YouthCan IMPACT project in particular, but also general findings that other integrated youth service hub stakeholders can consider to optimize their implementation and development processes.

The startup process involved the formation of a team of project leads, youth, caregivers, community mental health and hospital clinicians and decisionmakers, researchers, the project coordinator, and other stakeholders to co-create and implement the YouthCan IMPACT initiative. Key facilitators during the early startup and implementation phases examined in this study included the development of the YCI-IYS model using a “bottom-up” co-design approach with built-in adaptability as well as its capacity to address an important issue with an evidence-supported approach that was perceived to have advantages over existing practices, while at the same time not being perceived as too complex. In addition, the initiative was happening in a broader context of positive provincial and national momentum regarding integrated youth services and the team involved key members that had boundary-spanning roles. Within the partner organizations and the YouthCan IMPACT team, the governance model with decentralized decision-making, strong individual self-efficacy, positive problem-solving attitudes (“yes” approach), “managerial patience,” expertise in youth and family engagement, high social capital, and the positive previous working relationships among stakeholders facilitated implementation success. Lastly, collaboration and team processes that were perceived as enabling and empowering, as well as comprehensiveness in implementation planning, allowed the model to be implemented in ways that fit with existing capacities. Reflection and adjustment throughout also facilitated engagement and ultimately implementation.

Key barriers to implementation included structural instability within some partner organizations, budgetary challenges, achieving fidelity across sites, and the initial limited integration of direct service staff in the development of the model. These barriers were overcome by effective leadership, relationship trust that allowed for honest discussion, and the involvement of operational managers in the implementation process. In sum, important factors captured by the CFIR framework supported the successful implementation of the YouthCan IMPACT initiative, including aspects of the inner and outer setting, intervention characteristics, individual characteristics, and process.28

There is increasing provincial, national, and international interest in developing and implementing integrated youth service hubs to address systems challenges. Integrated youth service like the YouthCan IMPACT model provides a wide range of coordinated services to youth (e.g., mental health services, physical health services, social services etc.) in a “one-stop shop” format.17 Similar models have been developed across Canada and around the world.14,15,16,17,18,19 Research has been conducted regarding various aspects of these service hubs, including youth profiles, service use, and outcomes.11, 12 Ongoing evaluation of additional aspects, including the implementation process, is critical to continuing to optimize processes in this growing systems transformation movement. CFIR has been used to conduct formative evaluations of diverse health service implementation initiatives.56,57,58 However, to the best of our knowledge, the startup process and the barriers and facilitators involved in the development of integrated youth service hubs have not been documented using the CFIR framework.

An important question faced by the stakeholders in the integrated youth service hub landscape is the sustainability of these hub models. Each model has its own funding structure, including government health funding and philanthropic funding, in some cases supplemented by research funding. The YCI-IYS model aims to leverage existing services whenever possible. By co-designing the model with key stakeholders to address widely recognized system and service gaps affecting these stakeholders, and by restructuring existing services through creative partnerships to optimize service delivery rather than building new services, sustainability was enhanced from the outset. As research findings continue to emerge supporting the effectiveness of such models, it will be important that policymakers recognize the benefits of integrated youth service hub models and ensure that the funds are available to support them on an ongoing basis.

Based on our findings from the startup phase of the YouthCan IMPACT project, we suggest that it is feasible and effective to tackle the deficits in the current mental health system with innovative and collaborative efforts. This project confronts the deficits in youth mental health system by collaborating with existing services rather than replacing them by a new set of services.12 Although implementation of a highly collaborative project may face barriers,59,60,61,62 these barriers can be overcome through strong leadership and stakeholder involvement throughout the project phases, thus optimizing the implementation process. The current findings go hand-in-hand with a YouthCan IMPACT Implementation Guidebook,63 which aims to support teams interested in implementing community-based integrated service hub models of youth mental health service delivery.

Limitations

Limitations of this study include the possibility that social desirability could have affected participants’ responses during interviews and surveys. This was mitigated by having an unfamiliar research staff member conduct the interviews and review surveys for identity-revealing information and anonymize these responses. In addition, this study describes the startup process of an integrated youth services model with heavy research involvement and in an urban setting. The same process may not be applicable to developing hub-like models in other contexts (non-research) or in other settings (e.g., rural settings). Research results on the effectiveness of the model are not yet available, although work is in progress.12

Implications for Behavioral Health

Understanding the process of implementing an integrated youth service hub is crucial, since the system is moving toward this type of holistic approach for youth mental health and substance use concerns.11,12,13 This formative evaluation explores the potential barriers and facilitators of successful implementation of an integrated youth services model of service delivery. It can be used by existing, as well as future, hubs to understand potential barriers and facilitators to implementation, including the importance of positive partnerships and persistence. Involvement of service users beginning during early development and through implementation helps not only in engaging them, but also to ensure services that are designed to address their needs. Evaluations, such as this formative evaluation of YouthCan IMPACT’s start-up and early implementation phases, are essential for enhancing implementation of services for youth with mental health and addictions concerns.

References

  1. Osgood DW, Foster EM, Courtney ME. Vulnerable populations and the transition to adulthood. The Future of Children. 2010;20(1):209–229.

    PubMed  Article  Google Scholar 

  2. Waddell C, Shepherd C, Schwartz C, et al. Child and youth mental disorders: Prevalence and evidence-based interventions. Simon Fraser University, Faculty of Health Sciences. 2014.

    Google Scholar 

  3. Lopez AD, Mathers CD, Ezzati M, et al. Global burden of disease and risk factors. World Bank, Washington. 2006.

    Book  Google Scholar 

  4. Kim-Cohen J, Caspi A, Moffitt TE, et al. Prior juvenile diagnoses in adults with mental disorder: Developmental follow-back of a prospective-longitudinal cohort. Archives of General Psychiatry. 2003;60(7):709–717.

    PubMed  Article  Google Scholar 

  5. Statistics Canada. Leading causes of death, total population, by age group and sex [Table 102-0561] Canada, CANSIM (database). 2015; Available online at http://www5.statcan.gc.ca/cansim/a26?lang=eng&id=1020561. Accessed February 8, 2017.

  6. Tylee A, Haller DM, Graham T, et al. Youth-friendly primary-care services: How are we doing and what more needs to be done? The Lancet. 2007;369(9572):1565–1573.

    Article  Google Scholar 

  7. Kozloff N, Cheung AH, Ross LE, et al. Factors influencing service use among homeless youths with co-occurring disorders. Psychiatric Services. 2013;64(9):925–928.

    PubMed  Article  Google Scholar 

  8. Parents for Children’s Mental Health. The position of families on transformation of child and youth mental health system. 2013; Available online at https://www.pleo.on.ca/wp-content/uploads/2013/12/The-Position-of-Families-on-Transformation.pdf, Accessed February 8, 2017.

  9. Viswanathan M, Ammerman A, Eng E, et al. Community-based participatory research: Assessing the evidence: Summary. Evidence Report/Technology Assessment. 2004;99:1–8.

    Google Scholar 

  10. Settipani C, Cleverley K, Hawke Ld, et al. Essential components of integrated care for youth with mental health and addiction needs: Protocol for a scoping review. BMJ Open. 2017;7(4):e015454.

    PubMed  PubMed Central  Article  Google Scholar 

  11. Hetrick SE, Bailey AP, Smith KE, et al. Integrated (one-stop shop) youth health care: Best available evidence and future directions. Medical Journal of Australia. 2017;207(10):S5-S18.

    PubMed  Google Scholar 

  12. Henderson JL, Cheung A, Cleverley K, et al. Integrated collaborative care teams to enhance service delivery to youth with mental health and substance use challenges: Protocol for a pragmatic randomised controlled trial. BMJ Open. 2017;7(2):e014080.

    PubMed  PubMed Central  Article  Google Scholar 

  13. McGorry P, Bates T, Birchwood M. Designing youth mental health services for the 21st century: Examples from Australia, Ireland and the UK. The British Journal of Psychiatry. 2013;54(s54):s30–35.

    PubMed  Article  Google Scholar 

  14. BC-IYSI Working Group. British Columbia integrated youth services initiative: Rationale and overview. British Columbia, Canada. 2015.

  15. Malla A, Iyer S, McGorry P, et al. From early intervention in psychosis to youth mental health reform: A review of the evolution and transformation of mental health services for young people. Social Psychiatry and Psychiatric Epidemiology. 2016;51(3):319–326.

    PubMed  Article  Google Scholar 

  16. Ontario Ministry of Health and Long Term Care. Ontario Providing Faster Access to Mental Health Services for Thousands of People. 2017; Available online at https://news.ontario.ca/mohltc/en/2017/02/ontario-providing-faster-access-to-mental-health-services-for-thousands-of-people.html. Accessed March 18, 2019.

  17. McGorry P, Tanti C, Stokes R, et al. headspace: Australia’s National Youth Mental Health Foundation - where young minds come first. Medical Journal of Australia. 2007;187(7):S68.

    PubMed  Google Scholar 

  18. Illback RJ, Bates T, Hodges C, et al. Jigsaw: Engaging communities in the development and implementation of youth mental health services and supports in the Republic of Ireland. Journal of Mental Health. Oct 2010;19(5):422–435.

    PubMed  Article  Google Scholar 

  19. Zonneveld R. A one-stop shop for Wellington youth. Kai Tiaki: Nursing New Zealand. 2014;20(7):28.

    Google Scholar 

  20. Kadvany E. State approves $15M for youth mental health centers. Mountain View Voice. 2018; Available online at https://www.mv-voice.com/news/2018/08/23/state-approves-15m-for-youth-mental-health-centers. Accessed March 18, 2019.

  21. Ministère de santé et services sociaux. Santé mentale chez les jeunes de 12 à 25 ans – Le Gouvernement du Québec lance le projet Aire ouverte. 2018; Available online at http://www.fil-information.gouv.qc.ca/Pages/Article.aspx?idArticle=2608134517. Accessed March 18, 2019.

  22. Henderson JL, Hawke LD, Relihan J. Youth engagement in the YouthCan IMPACT trial. Canadian Medical Association Journal. 2018;190(Suppl):S10-S12.

    PubMed  Article  Google Scholar 

  23. Hawke LD, Mehra K, Settipani C, et al. What makes mental health and addictions services youth friendly? A scoping review (Accepted manuscript). BMC Health Services Research. 2019;19(257):2–16.

  24. Perkins R. The effectiveness of one session of therapy using a single-session therapy approach for children and adolescents with mental health problems. Psychology and Psychotherapy: Theory, Research and Practice. 2006;79(2):215–227.

    Article  Google Scholar 

  25. Bond C, Woods K, Humphrey N, et al. Practitioner review: The effectiveness of solution focused brief therapy with children and families: A systematic and critical evaluation of the literature from 1990–2010. Journal of Child Psychology and Psychiatry. 2013;54(7):707–723.

    PubMed  Article  Google Scholar 

  26. Kliem S, Kröger C, Kosfelder J. Dialectical behavior therapy for borderline personality disorder: A meta-analysis using mixed-effects modeling. Journal of Consulting and Clinical Psychology. 2010;78(6):936–951.

    PubMed  Article  Google Scholar 

  27. Hoffman PD, Fruzzetti AE, Buteau E, et al. Family connections: A program for relatives of persons with borderline personality disorder. Family Process. 2005;44(2):217–225.

    PubMed  Article  Google Scholar 

  28. Damschroder LJ, Aron DC, Keith RE, et al. Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science. 2009;4(1):50.

    PubMed  Article  Google Scholar 

  29. Grol R. Successes and failures in the implementation of evidence-based guidelines for clinical practice. Medical Care. 2001;39(8):II-46.

    CAS  Google Scholar 

  30. Stetler CB, Legro MW, Wallace CM, et al. The role of formative evaluation in implementation research and the QUERI experience. Journal of General Internal Medicine. 2006;21 Suppl 2(S2):S1–8.

    PubMed  PubMed Central  Article  Google Scholar 

  31. Doerr M, Edelman E, Gabitzsch E, et al. Formative evaluation of clinician experience with integrating family history-based clinical decision support into clinical practice. Journal of Personalized Medicine. 2014;4(2):115–136.

    PubMed  PubMed Central  Article  Google Scholar 

  32. Connell LA, McMahon NE, Harris JE, et al. A formative evaluation of the implementation of an upper limb stroke rehabilitation intervention in clinical practice: A qualitative interview study. Implementation Science. 2014;9(1):90.

    PubMed  Article  Google Scholar 

  33. Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program using the Consolidated Framework for Implementation Research (CFIR). Implementation Science. 2013;8(1):51.

    PubMed  Article  Google Scholar 

  34. Henderson JL, Chaim G, Brownlie EB. Collaborating with community-based services to promote evidence-based practice: Process description of a national initiative to improve services for youth with mental health and substance use problems. Psychological Services. 2017;14(3):361–372.

    PubMed  Article  Google Scholar 

  35. Henderson JL, MacKay S, Peterson-Badali M. Closing the research-practice gap: Factors affecting adoption and implementation of a children’s mental health program. Journal of Clinical Child & Adolescent Psychology. 2006;35(1):2–12.

    Article  Google Scholar 

  36. Henderson JL, Mackay S, Peterson-Badali M. Interdisciplinary knowledge translation: Lessons learned from a mental health: Fire service collaboration. American Journal of Community Psychology. 2010;46(3–4):277–288.

    PubMed  Article  Google Scholar 

  37. Brink SG, Basen-Engquist KM, O'Hara-Tompkins NM, et al. Diffusion of an effective tobacco prevention program. Part I: Evaluation of the dissemination phase. Health Education Research. 1995;10(3):283–295.

    CAS  PubMed  Article  Google Scholar 

  38. Parcel GS, O'Hara-Tompkins NM, Harrist RB, et al. Diffusion of an effective tobacco prevention program. Part II: Evaluation of the adoption phase. Health Education Research. 1995;10(3):297–307.

    CAS  PubMed  Article  Google Scholar 

  39. Steckler A, Goodman RM, McLeroy KR, et al. Measuring the diffusion of innovative health promotion programs. American Journal of Health Promotion. 1992;6(3):214–224.

    CAS  PubMed  Article  Google Scholar 

  40. Heffernan OS, Herzog TM, Schiralli JE, et al. Implementation of a youth-adult partnership model in youth mental health systems research: Challenges and successes. Health Expectations. 2017;20(6):1183–1188.

    PubMed  PubMed Central  Article  Google Scholar 

  41. Rickwood DJ, Mazzer KR, Telford NR, et al. Changes in psychological distress and psychosocial functioning in young people visiting headspace centres for mental health problems. Medical Journal of Australia. 2015;202(10):537–542.

    PubMed  Article  Google Scholar 

  42. Rickwood DJ, Telford NR, Mazzer KR, et al. The services provided to young people through the headspace centres across Australia. Medical Journal of Australia. 2015;202(10):533–536.

    PubMed  Article  Google Scholar 

  43. Repper J, Carter T. A review of the literature on peer support in mental health services. Journal of Mental Health. 2011;20(4):392–411.

    PubMed  Article  Google Scholar 

  44. Ontario Ministry of Child and Youth Services. Moving On Mental Health: A system that makes sense for children and youth. 2012; Available online at http://www.children.gov.on.ca/htdocs/english/documents/specialneeds/mentalhealth/momh.pdf. Accessed February 8, 2017.

  45. Canadian Institutes of Health Research. Canada’s Strategy for Patient-Oriented Research: Improving health outcomes through evidence-informed care. 2011; Available online at http://publications.gc.ca/collections/collection_2013/irsc-cihr/MR4-27-2011-eng.pdf. Accessed February 8, 2017.

  46. Greenhalgh T, Robert G, Macfarlane F, et al. Diffusion of innovations in service organizations: Systematic review and recommendations. The Milbank Quarterly. 2004;82(4):581–629.

    PubMed  PubMed Central  Article  Google Scholar 

  47. Harris FM, Maxwell M, O’Connor RC, et al. Developing social capital in implementing a complex intervention: A process evaluation of the early implementation of a suicide prevention intervention in four European countries. BMC Public Health. 2013;13(1):158.

    PubMed  PubMed Central  Article  Google Scholar 

  48. Damanpour F. Organizational innovation: A meta-analysis of effects of determinants and moderators. Academy of Management Journal. 1991;34(3):555–590.

    Google Scholar 

  49. Pinto JK, Slevin DP. The project champion: Key to implementation success. Project Management Institute. 1989;20(4):15–20.

    Google Scholar 

  50. Shaw EK, Howard J, West DR, et al. The role of the champion in primary care change efforts: From the State Networks of Colorado Ambulatory Practices and Partners (SNOCAP). The Journal of the American Board of Family Medicine. 2012;25(5):676–685.

    PubMed  Article  Google Scholar 

  51. Edmondson AC, Bohmer RM, Pisano GP. Disrupted routines: Team learning and new technology implementation in hospitals. Administrative Science Quarterly. 2001;46(4):685–716.

    Article  Google Scholar 

  52. Chamberlin J. User/consumer involvement in mental health service delivery. Epidemiology and Psychiatric Sciences. 2005;14(1):10–14.

    Article  Google Scholar 

  53. Burns L, Bradley E, Weiner B. Shortell and Kaluzny’s healthcare management: Organization, design and behavior.Cengage Learning; 2011.

    Google Scholar 

  54. Gist ME, Mitchell TR. Self-efficacy: A theoretical analysis of its determinants and malleability. Academy of Management review. 1992;17(2):183–211.

    Article  Google Scholar 

  55. Hawke LD, Relihan J, Miller J, et al. Engaging youth in research planning, design and execution: Practical recommendations for researchers. Health Expectations. 2018;21(6):944–949.

    PubMed  PubMed Central  Article  Google Scholar 

  56. Kowalski CP, Veeser M, Heisler M. Formative evaluation and adaptation of pre-and early implementation of diabetes shared medical appointments to maximize sustainability and adoption. BMC Family Practice. 2018;19(1):109.

    PubMed  PubMed Central  Article  Google Scholar 

  57. Hahn EE, Munoz-Plaza CE, Schottinger JE, et al. Developing innovative models of care for cancer survivors: Use of implementation science to guide evaluation of appropriateness and feasibility. Support Care Cancer. Aug 24 2018.

  58. Hagedorn HJ, Brown R, Dawes M, et al. Enhancing access to alcohol use disorder pharmacotherapy and treatment in primary care settings: ADaPT-PC. Implementation Science. May 10 2016;11:64.

    PubMed  Article  Google Scholar 

  59. Weist MD, Lowie JA, Flaherty LT, et al. Collaboration among the education, mental health, and public health systems to promote youth mental health. Psychiatric Services. 2001;52(10):1348–1351.

    CAS  PubMed  Article  Google Scholar 

  60. Alkema GE, Shannon GR, Wilber KH. Using interagency collaboration to serve older adults with chronic care needs: The Care Advocate Program. Family & Community Health. 2003;26(3):221–229.

    Article  Google Scholar 

  61. Lewandowski C, GlenMaye L. Teams in child welfare settings: Interprofessional and collaborative processes. Families in Society: The Journal of Contemporary Social Services. 2002;83(3):245–256.

    Article  Google Scholar 

  62. Robillard AG, Gallito-Zaparaniuk P, Arriola KJ, et al. Partners and processes in HIV services for inmates and ex-offenders: Facilitating collaboration and service delivery. Evaluation Review. 2003;27(5):535–562.

    PubMed  Article  Google Scholar 

  63. Henderson JL, Hawke LD, Darnay K, et al. Building Integrated Service Hub Models for Youth Mental Health and Addictions: The YouthCan IMPACT Experience and Guidebook. Toronto, Canada: Centre for Addiction and Mental Health. 2018; Available at: https://is.gd/YouthCanIMPACTGuidebook.

Download references

Funding

Funding was provided by the Ontario SPOR SUPPORT Unit, which is supported by the Canadian Institutes of Health Research and the Province of Ontario. The project was also made possible through a financial contribution from the Margaret and Wallace McCain Centre for Child, Youth and Family Mental Health, Centre for Addiction and Mental Health (CAMH). The funders play no role in the design of the review or the collection, analysis, and interpretation of the data.

Author information

Affiliations

  1. Margaret and Wallace McCain Centre for Child, Youth and Family Mental Health, Centre for Addiction and Mental Health, Toronto, ON, Canada

    Joanna Henderson PhD, CPsych, Margaret Hess BSc, Kamna Mehra MSc, DNB Psychiatry & Lisa D Hawke PhD

  2. Department of Psychiatry, University of Toronto, Toronto, ON, Canada

    Joanna Henderson PhD, CPsych, Margaret Hess BSc & Lisa D Hawke PhD

Authors
  1. Joanna Henderson PhD, CPsych
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Margaret Hess BSc
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kamna Mehra MSc, DNB Psychiatry
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lisa D Hawke PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joanna Henderson PhD, CPsych.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Disclaimers

None.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Henderson, J., Hess, M., Mehra, K. et al. From Planning to Implementation of the YouthCan IMPACT Project: a Formative Evaluation. J Behav Health Serv Res 47, 216–229 (2020). https://doi.org/10.1007/s11414-019-09658-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11414-019-09658-4