Background

A significant body of evidence supports the importance of evidence-based practice (EBP) in community mental health and health settings [1], yet these practices are largely underused by organizations and clinicians [2,3,4]. Implementation strategies are the interventions used to increase adoption, implementation, and sustainment of EBP in health services [5]. The evidence base for implementation strategies has advanced significantly in recent years including the identification of a complex set of factors related to implementation success. These factors include organizational- and individual-level factors, as well as characteristics related to the intervention, and the economic, political, and social context [6,7,8]. Although there are several proposed typologies of implementation strategies [5], no systematic approach to developing these strategies has been proposed [9]. Moreover, implementation strategy development has rarely systematically incorporated stakeholder involvement in the design [10], a critical oversight given research in healthcare suggesting that stakeholder input results in more effective outcomes [11, 12]. These weaknesses may explain why many implementation strategies fail to improve either implementation or clinical outcomes and why most strategies fail to engage their targeted mechanisms of action [13].

The importance of participatory approaches has been long recognized [14]. Multiple research traditions and disciplines such as implementation science, community-partnered research, and action research have incorporated provider input through stakeholder engagement methods and dynamic partnership [14,15,16,17,18]. Participatory design approaches are lauded for creating effective implementation strategies because they provide systematic methods to include and empower stakeholders at the beginning stages [19,20,21], and provide opportunities for stakeholders to be involved in design [14]. The current study utilizes an innovation tournament, a structured participatory design approach, in community mental health, as a springboard for implementation strategy co-design. Innovation tournaments are a form of crowdsourcing in which a host issues a call for ideas to address a specific challenge or problem within a system, and frontline providers and staff who work within the system are invited to submit their ideas for how to address the challenge. After multiple steps of screening and evaluation, a few ideas emerge as winners and others do not advance. Systematic review evidence points to the effectiveness of such crowdsourcing participatory design strategies in generating novel and useful solutions to complex and intractable problems in areas outside of healthcare settings [22,23,24].

The aims of the current study were twofold: (a) to test the utility of an innovation tournament for developing participatory-informed implementation strategies that enhance the use of evidence-based practices (EBPs) within a large behavioral health system and (b) to generate ideas from clinicians about the best ways for organizations to facilitate EBP implementation. Our study adds to the literature in several ways. First, our study extends the current literature on participatory approaches in implementation science, specifically providing proof-of-concept of the utility of the innovation tournament approach. We are aware of only one other study that applied the innovation tournament methodology to problems in healthcare systems and that study included a single organization focused on improving patients’ experience of care [24]. Second, we extend this work by using an innovation tournament to address EBP implementation and by applying the method across an entire behavioral health system comprised of a network of 200 independent treatment organizations (of mental health and substance use disorders) which are affiliated only to the extent that they are subject to the same policy and funding constraints through a single policy and payer infrastructure. Third, we collected ideas directly from the stakeholders who will be targeted by implementation strategies and have the most intimate understanding of factors that could affect the implementation process. The innovation tournament provided a structured and efficient way to integrate insights from clinicians (who have unique expertise about the problem), system leaders (who can speak to feasibility and financing), and experts in behavioral economics who understand drivers of implementation-relevant behavior. Behavioral science, including psychology and behavioral economics, can contribute valuable information about the best way to structure implementation efforts to optimize the design of implementation strategies that target organizations, critically important given the scarcity of resources in the mental health system.

Due to past successful experiences with recruitment within this system, we anticipated high engagement and a successful tournament. We had one hypothesis regarding the content of ideas generated in the tournament: In our past work, financial challenges such as the scarcity of resources in the mental health system and the organizational financial investment required to support EBPs have been highlighted by all stakeholders [6, 25]. In addition, clinicians have previously reported that they do not feel incentivized by their provider organizations to implement EBPs [26]. We hypothesized therefore that the majority of ideas offered by clinicians would be related to compensation and financial incentives.

Method

This study was approved by the Institutional Review Boards of the University of Pennsylvania and City of Philadelphia.

Setting

The Department of Behavioral Health and Intellectual disAbility Services (DBHIDS) in Philadelphia is a large publicly funded behavioral health system that annually services approximately 169,000 individuals with mental health and substance use disorders. Since 2007, DBHIDS has supported the implementation of multiple EBPs [6, 26, 27] including cognitive behavior therapy [28, 29], trauma-focused cognitive-behavioral therapy [30], prolonged exposure [31], dialectical behavior therapy [32], and parent-child interaction therapy [33]. DBHIDS funds training and consultation in line with treatment developer recommendations, and staff time to coordinate implementation, training, and ongoing consultation by treatment developers for each of the five EBPs. In 2013, the Evidence-Based Practice and Innovation Center (EPIC) was launched as a centralized infrastructure for EBP administration. In addition to supporting the EBP initiatives, which predated the creation of this entity, EPIC aligned policy, fiscal, and operational approaches to EBP implementation by developing systematic processes to contract for EBP delivery, hosting events to publicize EBP delivery, designating providers as EBP agencies, and creating enhanced rates for the delivery of some EBPs [27].

Procedure

This innovation tournament procedure was borrowed from Terwiesch and colleagues [24], developed through the Penn Medicine Center for Health Care Innovation [34].

Recruitment

We invited, by email, publicly funded behavioral health organization leaders (n = 210), clinicians (n = 527), and other community stakeholders (e.g., leaders of community provider organizations, directors of a clinician training organization; n = 6) in Philadelphia to participate in an innovation tournament called “The Philly Clinician Crowdsourcing Challenge.” We also e-mailed the invitation to four local electronic mailing lists known to reach large swaths of the network (e.g., the managed care organization listserv, the community provider organization listserv). Leaders and stakeholders were asked to forward the email to clinicians, so it is impossible to know exactly how many clinicians received and read the request. Moreover, some clinicians may have received the invitation more than once (i.e., from a direct e-mail and from their executive director). We sent leaders and stakeholders a priming email 1 week prior to the start of the tournament [35]. We then sent an initial invitation e-mail containing the link to participate and four reminder emails over the course of 5 weeks. The innovation tournament link was live for 5 weeks from February to March 2018. Potential participants clicked on the link and provided consent.

Tournament platform

The “Your Big Idea Platform,” powered by the Penn Medicine Center for Health Care Innovation, is a web-based platform that enables crowdsourcing for ideas and solutions. Idea challenges are posted to the platform, and participants can submit an idea, rate other ideas on a 1–5 “star” scale, and comment and rate ideas submitted by other participants [24].

Idea elicitation prompt

The tournament prompt (i.e., the question posed to potential participants) was developed through an iterative process. The authors created an initial draft of the tournament prompt and solicited feedback from the project’s steering committee. Our team conducted four cognitive interviews with clinicians and policy-makers to receive feedback on the wording, score, and design of the prompt. The final prompt asked clinicians, “How can your organization help you use evidence-based practices in your work?” Participants could submit and rate as many ideas as they wished.

Idea vetting and selection of winners

We assembled an expert panel of stakeholders to vote on the ideas and determine the winners. We wanted the committee to represent the perspectives of those who (a) understand what is feasible and acceptable to clinicians, (b) are in a position to implement the ideas, and (c) can evaluate the idea’s potential to leverage principles of behavioral economics. Therefore, the Challenge Committee included two agency stakeholders, two city administrators (including the Commissioner of the Philadelphia Department of Behavioral Health and Intellectual DisAbility Services [DBHIDS]), and two behavioral science experts. To minimize burden on our Challenge Committee, the research team refined the list from 65 to 22 by eliminating ideas that were not actionable (e.g., increase inpatient stays to a minimum of 60 days) or too similar to another, better-elaborated idea. The committee was asked to rate the ideas based on their enthusiasm for the idea’s “potential to increase therapist use of evidence-based practice” using a 1–5 Likert scale (1 = not at all enthusiastic, 5 = very enthusiastic). The top six ideas as rated by the Challenge Committee were chosen as winners.

Participants who submitted winning ideas (hereafter referred to as “innovators”) were invited to a working lunch with our team to refine and develop their idea leveraging principles of behavioral economics. For example, innovators discussed how to amplify the impact of their proposed strategy using such behavioral economic principles as changing defaults, peer comparisons, loss-framed incentives, and nudges [36,37,38,39].

During this meeting, innovators prepared a brief presentation summarizing their idea for a community-facing event, The Idea Gala, held in May 2018. All who received the invitation to participate in the tournament (described above) were invited to the event. During The Idea Gala, each of the six innovators presented their idea for 3 min. Attendees had the option of using an online polling system (www.mentimeter.com) to answer the following question on a 1–10 Likert scale (1 = not at all excited, 10 = very excited): “How excited are you by this idea?” and to submit brief comments about the ideas. After the presentation of each idea, there were 10 min of questions from the audience and an unstructured discussion of the idea by the Challenge Committee. Challenge Committee members focused their comments on the problem, the solution, applications of principles of behavioral economics to enhance implementations, metrics for success, and resources necessary for implementation.

Coding and categorization of ideas

After reviewing submitted ideas, we developed a coding scheme to categorize the ideas into eight non-mutually exclusive categories. These categories included (a) training, (b) financing and compensation, (c) clinician support and preparation tools, (d) client support, (e) EBP-focused supervision, (f) changes to the scope or definition of EBPs, (g) changes to the system and structure of publicly funded behavioral health care, and (h) other. Two study team members (RS and VB) categorized all the ideas, and inter-rater reliability was found to be excellent (K = .92). Discrepancies were resolved through discussion.

Results

Tournament feasibility and respondent characteristics

A total of 65 ideas were submitted by 55 participants. The participants represented 38 behavioral health organizations representing multiple levels of care (e.g., outpatient, inpatient, residential) and specialties (mental health, substance use). The majority of participants were white (61.8%) and female (61.8%). Additionally, 59 participants submitted 899 ratings, and 22 participants submitted 55 comments on the 65 ideas. Of the clinicians (n = 527) who were contacted directly to participate in the tournament, 12.3% submitted an idea. According to personal communications with “Your Big Idea” staff, this response rate is consistent with (or exceeds) the average response rate (5%) of challenges administered through the “Your Big Idea” platform.

Categories of solutions

Submitted ideas encompassed all aspects of supporting EBPs. We categorized ideas into eight categories. Over 40% of ideas pertained to training (42%), followed by ideas related to changes to financing and compensation (26%), including paid training, financial incentives, and rewards. A large proportion of ideas included clinician support and preparation tools (22%) and supervision (17%). Other categories included supporting clients (5%), changes to the scope or definition of EBPs (8%), and changes to the system and structure of publicly funded behavioral health care (6%). See Table 1 for examples of ideas and ratings by category by other participants.

Table 1 Categories of solutions
Full size table

Winning ideas and The Idea Gala

See Table 2 for the Challenge Committee’s ratings of the top winning ideas, each innovator’s idea, and the target problem addressed by the idea.

Table 2 Top six innovator ideas
Full size table

A total of 85 participants (12 project staff, 6 Challenge Committee members, 6 innovators, 61 other attendees) representing 22 organizations attended The Idea Gala. Attendees’ ratings of the innovators’ ideas are shown in column 3 of Table 2, alongside the Challenge Committee’s ratings in column 2. Both groups rated “pre-session preparation,” defined as creating a relaxing waiting room to prepare clients for the upcoming session, the highest among the 6 winning ideas.

Discussion

The present study is the first to utilize an innovation tournament to crowdsource ideas from community clinicians about how they can best be supported to use EBPs. The main purpose and benefit of an innovation tournament is to engage a community through participatory design to provide their best ideas. While many studies have included stakeholder elicitation of suggestions, preferences, and targets for intervention (e.g., [40]), an innovation tournament has a unique contribution as a participatory method for the following reasons. First, an innovation tournament can engage an infinite number of people in a fun and creative way for low incremental cost, while giving each participant an equal voice with the submission of an idea. A tournament allows for quick identification of potentially high impact ideas at every level of a system and can provide a training ground to empower stakeholders’ good ideas and improve their ability to express them. Lastly, a tournament provides public acknowledgement of people who come up with ideas, and this may increase buy-in to the ideas when ultimately implemented (the “Ikea effect”) and encourage future participation. Innovation tournaments are therefore a relatively inexpensive and efficient way to solicit and powerfully engage end-user input.

Our tournament was a success in this regard: we had 65 ideas submitted by 55 participants and 899 ratings of ideas by those participants. The number of submissions and overall participation exceeded our expectations, and we engaged many more in our community through our celebratory event. The study team was continuously struck by the high level of community stakeholder enthusiasm throughout the study period. While use of crowdsourcing in health research is nascent [22], this study demonstrates how an innovation tournament can be not only feasible to implement across an entire system, but also a successful and acceptable means to engage a community of service providers in sharing their expertise to generate useful implementation strategies that address local barriers, contexts, and populations.

The prompt for our innovation tournament was purposefully broad in order to encourage clinicians to share their unique local knowledge about targeted barriers that interfere with EBP implementation and their innovative ideas for overcoming those barriers. We wanted to capture the full spectrum of levers that support clinicians in implementing EBPs [34]. The most common theme reflected in the ideas involved training. This was surprising given that over the last 10 years, DBHIDS has supported implementation of several EBPs in over 50 mental health and substance use provider agencies by funding training, consultation, coordination, administration through EPIC, and in some cases reimbursing for lost therapist time, in addition to providing an enhanced reimbursement rate [26, 27]. Despite this, clinicians within this system overwhelmingly reported that additional (free) training would increase their use of EBPs with their clients. Obviously, active training is an important vehicle to change therapist behavior; training also impacts knowledge and attitudes [41]. Nonetheless, training is not the answer to every implementation problem: even clinicians trained via current gold standard approaches (i.e., workshop, manual, and clinical supervision) often do not demonstrate fidelity [42].

One alternative and more parsimonious explanation for well-trained clinicians’ thirst for additional training is that, put simply, EBPs are hard. Almost every EBP requires a skilled interventionist to deliver a complex, multi-component repertoire of behaviors, while responding to the client’s inputs and needs. It is striking that two thirds of the ideas put forth in the tournament target training and/or clinician support tools. There is also evidence from the research literature suggesting that trained clinicians do not feel confident delivering EBPs [43]. If we start with an assumption that clinicians want to deliver EBPs and would like to feel confident that they can enact effective treatment, we might accelerate implementation if we invest in implementation strategies that make our treatments easier to do, rather than additional training initiatives. Future research on implementation strategy design might draw on insights from behavioral economics such as “nudges” and changing the choice architecture; simple checklists can make multistep procedures easier [44,45,46].

Consistent with our hypothesis that a significant portion of ideas would be financially motivated, over a quarter of the submitted ideas pertained to compensation and pay. As we found in our prior research, stakeholders note that EBPs are associated with higher marginal costs that need to be reimbursed and that existing reimbursement strategies rarely cover these higher costs [6, 25]. While there were ideas calling for additional lump compensation, there were relatively few ideas involving complex incentive structures such as those used in pay-for-performance or value-based payment models. This may reflect the “bottom up” approach of the innovation tournament; involving clinicians helped us identify new opportunities that would be overlooked if innovation was left to administrators and executives alone. Ultimately, a combination of “top down” and “bottom up” approaches may tell us most about how to motivate complex, expensive repertoires of behavior such as those required to implement EBPs.

There are limitations to the study design and methodology that deserve mention and highlight avenues for future research. First, because it draws from one system that is clearly aligned with promoting EBPs—this sample may not be representative of clinicians in this system or other public behavioral health systems. The specificity of implementation strategies that emerge from an innovation tournament conducted within a given service system is both an advantage and a limitation of this approach—the strategies developed through the tournament may be ideal for a particular context and system but may not generalize to other contexts and systems. Second, beyond “winning” the tournament, clinicians in our study were not promised that their ideas would be implemented. This may have restricted clinicians’ willingness to participate or encouraged ideas that were not “implementable.” Given evidence suggesting discordance between behavior and stated preferences [47], we framed tournament ideas as an “input” to design rather than relying entirely on stated preferences. Moving these inputs to design will require additional scholarship and analysis. Currently, we are re-analyzing the ideas emphasizing behavioral processes and barriers, not just elicited preferences, as part of a multifaceted approach towards implementation strategy design. Third, although our response rate was more than double that of the average innovation tournament, some tournaments have generated much higher participation rates, even in larger organizations. Future research should examine how participants’ expectations regarding the eventual implementation of their ideas influence engagement. Fourth, we elected to form a Challenge Committee of city administrators, agency stakeholders, and experts in behavioral economics to decide on the winners; a challenge committee comprised of different stakeholder groups (e.g., clinicians, patients) may have selected different strategies as winners. Future studies should compare whether different stakeholder groups select the same or different strategies and, most importantly, whether these choices predict the effectiveness of strategies used to improvement EBP implementation within the system.

Conclusion

Through a novel methodology for participatory design, findings from this study highlight the feasibility and utility of engaging clinicians—arguably the ultimate target of implementation strategies—through a structured, system-wide innovation tournament. Moreover, the innovation process does not have to end with the final and winning ideas [24]. Analysis of all of the ideas (or better yet, re-engagement with those who did not have winning ideas) can provide a fertile ground for future research. We believe the tournament succeeded in engaging stakeholders with critical expertise and provided valuable data that furthers the science of developing implementation strategies to improve the implementation of EBPs. More importantly, the success of our tournament was in engaging and empowering the community. Although we did not measure acceptability directly, the overwhelming enthusiasm for this project from clinicians, agency administrators, and city officials indicated to us that clinicians were pleased to be queried and felt validated that their ideas and viewpoints were not only heard, but celebrated. The clear implication to us is that involving clinicians in every stage of implementation (from strategy design to sustainability) consistent with community-participatory research is essential to designing effective implementation strategies that improve the quality of community-based care.