Learning Collaboratives: a Strategy for Quality Improvement and Implementation in Behavioral Health

Learning collaboratives are increasingly used in behavioral health. They generally involve bringing together teams from different organizations and using experts to educate and coach the teams in quality improvement, implementing evidence-based practices, and measuring the effects. Although learning collaboratives have demonstrated some effectiveness in general health care, the evidence is less clear in behavioral health and more rigorous studies are needed. Learning collaboratives may contain a range of elements, and which elements are included in any one learning collaborative varies widely; the unique contribution of each element has not been established. This commentary seeks to clarify the concept of a learning collaborative, highlight its common elements, review evidence of its effectiveness, identify its application in behavioral health, and highlight recommendations to guide technical assistance purveyors and behavioral health providers as they employ learning collaboratives to improve behavioral health access and quality.


Introduction
The last several decades have been marked by growing concerns about the quality of health care. A seminal report at the turn of the century by the Institute of Medicine highlighted that as many as 98,000 people were dying annually from medical errors. 1 This caught public attention and stimulated a national agenda for improving patient safety. What followed were efforts throughout health care, and behavioral health care, to further develop and disseminate evidencebased practices to enhance the quality and effectiveness of care. 2,3 Developing an evidence-base on safe and effective practices is an essential first step. However, translating science to practice and improving the quality of care delivered in real world settings has proved to be a daunting challenge. 4 While first developed in general health care, learning collaboratives, also called quality improvement collaboratives, have been used frequently in behavioral health by providers of training, technical assistance, and services as a strategy to promote quality improvement and implementation of a range of evidence-based practices (e.g., cognitive behavioral therapy for psychosis, integrated services for co-occurring mental health and substance use disorders, school mental health, supported employment, trauma informed care). [5][6][7][8][9][10][11][12] These collaboratives generally involve bringing together teams from different organizations and using experts to educate and coach them in a quality or implementation project and measure the effects. 13 Sharing of strategies, data, successes, and obstacles among participating teams is central to the approach. 14 With the proliferation of learning collaboratives has come wide variability in how they are conducted. [15][16][17] Technical assistance providers may be unclear about which components of a learning collaborative are most important, how long it should last, and when to offer a learning collaborative rather than an alternative improvement or implementation strategy (e.g., training workshop, community of practice). Behavioral health care providers may struggle with whether to invest significant staff, time, and financial resources in a learning collaborative, whether a collaborative being offered is adequately designed, and what outcomes to expect.
Learning collaboratives have been the subject of many publications and a more modest number of research studies. 6,7,16 This commentary stems from a project led by the Annapolis Coalition on the Behavioral Health Workforce for the Mental Health Technology Transfer Center Network to improve knowledge about collaboratives within the behavioral health community and to promote, if warranted, an increase in their effective use in the USA. The specific aims were to clarify the concept of a learning collaborative, highlight its common elements, review evidence of its effectiveness, identify its application in behavioral health, and highlight recommendations to guide technical assistance purveyors and behavioral health providers as they employ learning collaboratives to improve behavioral health access and quality.

Methods
This commentary integrates the findings from a traditional literature review of the published and gray literature and key informant interviews of experts in the field.

Literature search and review
A literature search of 16 terms related to learning collaboratives was conducted using Ovid MEDLINE, PsycINFO, AMED, PsycARTICLES, PsycEXTRA, Embase, CINAHL, Google Scholar, and Google with filters set for publications in English from 2010 to 2020. The search included the following terms: learning collaborative(s), quality improvement collaborative(s), quality improvement learning collaborative(s), quality collaborative(s), mental health service collaborative care, mental health service quality, behavioral health learning collaborative(s), behavioral health quality improvement collaborative(s), quality improvement collaborative(s) in behavioral health, evidence-based engagement strategy(ies), multidisciplinary quality improvement team(s), community based quality improvement, learning laboratory(ies), breakthrough series, and Network for the Improvement of Addiction Treatment (NIATx). Reference lists in the initial documents identified were reviewed to find additional resources. One member of the study team (MP) conducted the search and reviewed reference lists, which yielded 640 documents. He performed the initial analysis of this literature, identifying 151 relevant documents and then categorized them by focus (mental health, substance use, general health care, or integration of general health and behavioral health). A second member of the team (MH) conducted a detailed analysis of the relevant literature, identifying the major categories of findings and the national surveys, reviews, and individual studies that offered specific findings within those categories. Sources on learning collaboratives in general health care were examined first, providing a context for review of the more limited resources in behavioral health. The other members of the team then vetted the categorization scheme and the distillation of findings into those categories.

Key informant interviews
Semi-structured interview questions were developed after the review of the literature clarified topics needing further exploration. Fifteen interviews were conducted involving 17 individuals (see Acknowledgements for names) with substantial knowledge related to learning collaboratives in health or behavioral health, including those focused on diverse populations. Experience within this expert group included founding the learning collaborative movement; managing technical assistance and quality improvement networks; funding and shaping federal policy regarding technical assistance; and conducting research on learning collaboratives, quality improvement, and implementation. One member of the study team (MH) conducted, recorded, and analyzed the recordings, using open coding, which was then reviewed by another author (HG). 18 The first author (MH) integrated information from the literature review and interview findings. Another author (HG) shaped a conceptual framework that further organized the information into a set of distinct findings and recommendations. National leaders in the provision of behavioral health technical assistance, who comprised a Workforce Development Working Group for the Mental Health Technology Transfer Center Network, then reviewed and commented on a draft study report before its finalization.

Learning collaboratives: current knowledge
The following points describe and characterize current knowledge about learning collaboratives based on their extensive use in health care and increasing use in behavioral health, as well as evaluation of those efforts. They are organized by the specific aims of clarifying the concept of a learning collaborative, highlighting common elements and structure, reviewing evidence of effectiveness, and identifying applications in behavioral health.

Clarifying the concept of a learning collaborative
The traditional learning collaborative model has been reasonably well-defined, broadly disseminated, and widely adopted. Launched in 1995, the Institute for Healthcare Improvement (IHI) Breakthrough Series (BTS) model is considered the origin of the learning collaborative. 13 Key elements of a BTS collaborative, as identified by IHI in its early publications, included: selecting a specific improvement topic, recruiting expert faculty, enrolling organizations, providing face-to-face learning sessions, Plan-Do-Study-Act (PDSA) cycles of change, offering technical assistance to organization teams, inter-agency sharing and learning, and summation of results and lessons learned. 13,19,20 The IHI model became extremely popular and was widely disseminated and adopted nationally and internationally. 13,17 Currently, the term learning collaborative is often used loosely as a label for efforts at quality improvement and implementation that are often not well-defined or described. As learning collaboratives have proliferated, the term has been used for projects that vary widely from the original BTS model, including different elements, lengths, and frequency of meetings, raising questions about the use of the term and expected efficacy of the project. 15,16 Confusion and variation stem from a number of sources, including a lack of detail in published studies and descriptions of learning collaboratives. 6 Variation also occurs as technical assistance providers experiment with collaboratives that are shorter, less intensive, and more virtual to reduce overall cost and burden (Lang, Venkatesh, personal communication). 21 Technical assistance providers depart from the BTS model when the evidence-base for improving quality is not strong, funding is inadequate, the capacity of participating organizations is a concern, or the project goal is limited to provider skills (Lang, Reid, Venkatesh, personal communication). Finally, it is important to note that a BTS learning collaborative is distinct from learning communities and communities of practice, which tend to be more about individual provider development than organizational change, and more about learning than the collection and analysis of data. [25][26][27]

Common elements and functions
Experts have highlighted the elements of learning collaboratives they deem most important, though limited research has yet to consistently demonstrate the relative impact of any specific element. 20,28-30 A learning collaborative is a bundle of specific elements, such as the composition of the participating teams, number and format of learning sessions (in-person and/or virtual), use of different types of pre-work, and collection of and feedback about data. A systematic review of 20 studies of learning collaboratives in general health care identified 14 process and structural elements (see Table 1). 28 Although some elements were used more consistently than others (i.e., in-person learning sessions, PDSA cycles, and collection of data for quality improvement were each used by at least 15 of the 20 studies), research has not disentangled the effectiveness of each element.
The length of learning collaboratives varies greatly, and their optimal length and intensity have not been demonstrated. This issue is controversial, particularly as shortening the experience saves costs and reduces burden on participating agencies and their teams. IHI BTS collaboratives currently last between 18 and 24 months, whereas the average for behavioral health collaboratives, based on a systematic review by Nadeem and colleagues, was 14 months (Laderman, personal communication). 6 Factors to be considered in setting the length include complexity of the intervention, resources available to the sponsor and participating organizations, and goals to be achieved (Everett, Hoover, personal communication). On the one hand, organizational level change takes time and brief learning collaboratives may limit effectiveness and sustainability (Amaya-Jackson, Becker, Laderman, Reid, personal communication). On the other hand, traditional, lengthy collaboratives may outstrip the resources and attention span of many behavioral health agencies (Gustafson, personal communication).

Evidence-base for learning collaboratives
In general health care, research on traditional learning collaboratives, albeit incomplete, suggests some effectiveness in improving provider practices and health outcomes. 5,7,15-17 A systematic review of studies of BTS-style collaboratives from 1995 through 2014 found that most studies reported significant improvement on at least one clinical process or patient outcome measure with a wide range in the magnitude of changes. 7 More successful collaboratives addressed straightforward aspects of treatment and the large gap between current practice and the evidence-based practice being implemented. However, the authors noted that many studies were not included in the review due to flaws in design, lack of fidelity to a specific collaborative model, and sparse descriptions of collaborative elements. When other reviews have included a broader variety of learning collaboratives and quality improvement processes, they have found more mixed results and suggest that effectiveness varies by contextual factors; for example, learning collaboratives may be more effective in changing provider rather than patient outcomes, and effectiveness increases when a learning collaborative is paired with other quality improvement or implementation strategies. 5,[15][16][17] The more that learning collaboratives depart from the fidelity of the traditional BTS model, the less assured one can be of their effectiveness.
Organizational and system change is a complex, difficult process. Multiple reviews in health care and the field of dissemination and implementation science have shown that passive implementation strategies (e.g., distributing published or printed recommendations) and even traditional training strategies (e.g., workshops, lectures) are not effective in changing provider behavior and/or implementing evidence-based practices. [37][38][39] Moreover, using a single implementation strategy to foster practice change is less effective, in general, than multi-component strategies. [40][41][42] As is highlighted in Table 1, a learning collaborative is a multi-component strategy. Some research and strong consensus among experts suggest that Table 1 Learning collaborative process and structural elements * Adapted from Nadeem and colleagues 28 Learning collaborative element In person learning sessions Plan-Do-Study-Act (PDSA) cycles Site collection of new data for quality improvement (QI) Multidisciplinary QI teams QI team calls involving multiple teams Email or Web support Leadership involvement or outreach External support with data synthesis and feedback Site review of data and use of feedback Training for non-QI team staff by the QI team Pre-work: Convening an expert panel Pre-work: Organizations demonstrate commitment Training of non-QI team staff by experts Length of the collaborative learning collaboratives are likely more effective in improving provider practices and patient outcomes than low intensity interventions such as lectures, workshops, and webinars (Amaya-Jackson, Becker, Gustafson, personal communication). 30,43,44 A few studies comparing learning collaboratives to less intensive or single strategies have found promising results, and several larger scale, controlled trials are in process. 30,[43][44][45][46] Learning collaboratives in behavioral health In behavioral health, learning collaboratives have been used frequently to address a broad range of topics, including large scale, high-quality applications of the traditional BTS style model. Behavioral health learning collaboratives are described in over 60 publications which report on their perceived value in improving quality and implementing evidence-based practices, with several prominent examples. Over 50 collaboratives on Trauma Focused Cognitive Behavioral Therapy have been conducted by the National Center for Child Traumatic Stress. 4 The National Center for School Mental Health adapted the BTS model to improve school mental health systems across 14 states. 9 The Child Health and Development Institute used the BTS model in 15 collaboratives to disseminate and sustain various evidence-based mental health treatments. 47 Others have focused on behavioral health and primary care integration, supported employment, addiction treatment, and emergency department care. [48][49][50][51][52] Generally, learning collaboratives used for behavioral health topics, compared to health settings, are not markedly different, although those included in Nadeem and colleagues' systematic review tended to be shorter in length. 6 Few controlled studies have evaluated the effectiveness of learning collaboratives in behavioral health, though the body of research is growing. 51,53-55 Nadeem and colleagues' systematic review found that among the 20 studies that met review criteria, numerous positive trends were reported on provider and patient outcomes, sustainability, and acceptability of collaboratives to providers. 6 However, the absence of detailed descriptions of studies and adequate comparison data made it impossible to draw firm conclusions about the effects of learning collaboratives in behavioral health. It appears that the amount and quality of research on behavioral health learning collaboratives is increasing, offering the prospect of improved evidence about their effectiveness.
Health equity has received little attention among learning collaboratives in either general health care or behavioral health. 56,57 Serious inequities in access and quality of services related to race/ethnicity, gender identity, sexual orientation, income, and other social determinants of health are widely recognized in behavioral health care. [58][59][60][61] Several SAMHSA-funded, disparities-focused technical assistance centers offer a range of training and resources on services for diverse populations, including the National Hispanic and Latino and National American Indian and Alaska Native Mental Health Technology Transfer Centers, Behavioral Health Centers of Excellence on African-American, Aging, and LGBTQ populations, and National Network to Eliminate Disparities in Behavioral Health. 62 It is becoming seen as an imperative that organizational and policy changes be enacted to achieve behavioral health equity. 63,64 Recommendations Drawing on the appraisal of literature and key informant interviews, the following recommendations are offered for organizations that provide or receive technical assistance. The recommendations focus on the underlying rationale for choosing and using a learning collaborative for any given improvement or implementation initiative, the specifics of how a learning collaborative is conducted, including elements and structure, and issues of evaluation. General recommendations that apply to any improvement or implementation initiative appear first, followed by recommendations specific to learning collaboratives. In addition, Table 2 presents selected best practices in planning and conducting collaboratives summarized from the appraisal of the literature and interviews of experts.

For any quality improvement or implementation initiative
Adopt an explicit framework for clarifying goals and selecting the quality improvement or implementation strategy to accomplish those goals. The goals of a quality improvement or implementation project vary, and can include changes in attitudes, knowledge, skills, clinical practice, quality, and/or patient outcomes. The providers and recipients of technical assistance should use an explicit planning framework to clarify their goals (Laderman, personal communication). As one example, the Kirkpatrick Model was developed in the 1950s to promote effective training and has been used extensively in health care education. 72 It identifies four possible goals: Reaction-whether participants find the experience favorable, engaging, and relevant; Learningwhether participants acquire the intended knowledge, skills, attitudes, confidence, and commitment; Behavior-whether participants apply what they learned; and Results-whether targeted outcomes occur as a result of the intervention. 72 After clarifying goals, decisions can be made about which process improvement or implementation strategy should be used to achieve those goals (i.e., a training workshop may be appropriate for goals of reaction or learning, whereas a learning collaborative may be more appropriate if the goals are behavior or results). Other frameworks include concept mapping and group model building. 73 Ensure that actual improvements in health care quality and health outcomes are among the goals. Although lectures, workshops, and webinars can increase awareness, facilitate learning, and shift attitudes, more intensive, multi-component interventions, like learning collaboratives, are likely necessary to change provider practice and improve patient outcomes. 30,43,44 Improving health care and patient health are the ultimate goals of quality improvement and implementation, and should be well represented among the project goals, especially as the evidence regarding whether learning collaboratives affect change at the patient level is mixed. 5,[15][16][17] Adopt a model for improvement or logic model that explicitly identifies how planned changes will lead to desired outcomes and how those outcomes will be measured. While the selection of goals occurs at a broad conceptual level, detailed planning is needed to map out the process of change. One example is IHI's Model for Improvement that guides the technical assistance or service provider through this process. 69 Other models include Lean, Six Sigma, and NIATx (developed specifically for behavioral health). [74][75][76] Another example is a logic model, which is a widely recognized method for visually mapping the link between resources applied, activities planned, and desired outcomes and impact. The W.K. Kellogg Foundation offers a Logic Model Development Guide as a resource for non-profit organizations. 77 Be guided by the evidence on how to promote improvement and implementation, but do not be paralyzed by imperfections in the evidence. Professional perspectives about the effectiveness of learning collaboratives differ, which came to light during expert interviews. Technical assistance providers highlighted the practical and immediate need to choose among the best available strategies to improve quality and accelerate the adoption of evidence-based practices. Researchers emphasized the imperative for more clarity and fidelity to learning collaborative models and more sophisticated evaluations of their effects. In its current, imperfect state, the body of evidence points technical assistance and service providers toward change strategies, such as learning collaboratives, that are more likely to be effective, though still must be tailored to the unique organizational contexts in which they are applied. 30,43,44 Make health equity the lens through which all work on improvement and implementation is conducted. Adoption of this recommendation is long overdue. In the words of one expert, "Improvement is not occurring if whole groups of people are being left behind." 25 Learning collaboratives can be organized and focused on health equity projects and outcomes, as well as include a health equity perspective in all aspects of their development and evaluation. [63][64][65] Multiple frameworks and guides exist to assist health and behavioral health agencies to improve health equity. 56,65,66 Other specific recommendations related to health equity are listed in Table 2.
Measure the impact of initiatives by examining participant satisfaction, practice change, patient outcomes, cost benefit, and sustainability. A thorough evaluation should measure a variety of outcomes to understand what was done and whether it worked. Relevant outcomes in learning collaborative studies include patient and provider variables, acceptability of the collaborative model to providers, sustainability of the changes achieved, impact of specific elements of the collaborative, and estimates of the cost of the collaborative. 6 In addition, the Reach, Efficacy, Adoption, Implementation, and Maintenance (RE-AIM) framework is frequently used to organize outcomes at both the patient/ intervention level (e.g., Reach refers to how many patients receive the practice) and organization/ implementation level (e.g., Adoption refers to how many providers/organizations use the practice). 67

For learning collaboratives specifically
Offer or participate in a traditional IHI BTS learning collaborative for improvement or implementation if there are clear evidence-based practices to be adopted, and sufficient resources, provider interest, and capacity to make this approach viable. While traditional learning collaboratives involve significant investments of money, time, and effort, meaningful change generally requires significant investment (Nadeem, personal communication). If the pre-conditions of an evidence base, resources, interest, and capacity are not met, then a learning collaborative, in its traditional format, may not be the best option.
Make sure that the learning collaborative includes key elements identified by IHI and in systematic reviews of research on collaboratives. Individual elements of learning collaboratives have been identified in theoretical and empirical analyses (see Table 1). 15,28,30 While the unique impact of these elements has yet to be established, some elements are frequently used, and are well-described in related materials and resources. 13,68,69 When planning and considering participation in a learning collaborative, purveyors and providers should carefully consider which elements are included. 30 Consider offering or participating in variants of a learning collaborative, like learning networks or learning communities, if the evidence-base is less clear or resources and provider interest are less robust. Learning networks, learning communities, and communities of practice are generally less rigorous, intensive, and data-driven, with a more uncertain link to patient outcomes. [25][26][27]70 However, they are widely used, particularly to heighten awareness, promote shared learning, and increase professional development among individual practitioners or agencies with a common interest or focus. The impact of such activities on practice change and patient outcomes is less clear.
Pair time-limited collaboratives with participation in ongoing networks to enhance sustainability. Learning collaboratives promote organization and system change to help sustain improved outcomes over time. A key contribution in the field of behavioral health has involved combining time-limited learning collaboratives to catalyze change with ongoing strategies and structures, such as learning communities, peer networks, and technical assistance centers, to sustain change. These typically function to provide continuous training and technical support, centralized data collection, benchmarking, and peer learning among participating organizations. Exceptional examples are found in the areas of child trauma (The National Child Traumatic Stress Network), evidence-based practices in children's mental health (Child Health and Development Institute), school-based mental health (National Center for School Mental Health), and supported employment (The Individual Placement and Support [IPS] Employment Center). 4,9,47,50 Build capacity to offer or participate in collaboratives through internal staff development and partnerships with other organizations. Organizations can be strengthened by developing the competencies of their staff to offer or participate in learning collaboratives. Such staff development builds organizational capacity to set goals, develop strategic plans, organize change management teams, create logic models, conduct tests of change, implement evidence-based practices, improve quality, and measure outcomes. Several organizations offer guides to conducting collaboratives, compilations of change strategies, and PDSA resources. For example, IHI developed the Breakthrough Series College, IHI Open School, conferences, Improvement Coach professional development program, and a quality improvement toolkit. Other examples include the TOOLCIT Curriculum for Learning Collaborative Facilitators by the National Child Traumatic Stress Network, which includes an e-learning course, and Planning and Implementing a Successful Learning Collaborative guide by the New York Department of Health AIDS Institute. 68,69 Educational and training programs (such as the IHI College) are accessible ways for staff to enhance their knowledge and skills, as are the many other guides, resources, and experiential opportunities related to learning collaboratives. Partnering with other organizations to create and offer collaboratives is often an efficient way to build on internal capacities. Document learning collaboratives and lessons learned from them in detail to maximize the value of these efforts to the field. For the knowledge base about learning collaboratives to grow, especially in behavioral health, it is imperative that those who offer and participate in quality improvement and implementation efforts document the experience and disseminate that information. This includes providing detail about the collaborative elements used, the processes implemented, and the outcomes achieved. Several sets of guidelines for such documentation exist, including "Name it, Define it, Specify it" in the field of implementation science, and the Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0) from the field of health care improvement. 71,72 Two learning collaborative-specific strategies include the Agency for Healthcare Research and Quality's learning collaborative taxonomy and the template for describing any learning collaborative across 14 common dimensions developed by Nadeem et al. 6,73,74

Implications for Behavioral Health
The behavioral health field has yet to adequately bridge the enormous gaps in access to highquality, effective treatment for mental health and substance use disorders. 75,76 This is especially true for diverse populations. [59][60][61] Quality improvement and implementation of evidence-based practices are two related approaches being used to close these gaps. Learning collaboratives have emerged as a frequently employed strategy to assist with these efforts.
As learning collaboratives have proliferated, deviations from the traditional model have become commonplace, introducing confusion about the components of collaboratives and their effects. This commentary summarized information from the literature with wisdom drawn from interviews of experts to support technical assistance providers and behavioral health agencies in better understanding what learning collaboratives are and how to use them most effectively.
While some evidence suggests that the traditional model can change provider practice and improve patient outcomes, more research is needed to determine learning collaboratives' effects in behavioral health care, their essential elements, and optimal length. However, technical assistance providers and behavioral health agencies should not shy away from learning collaboratives because of imperfections in the evolving research. They can draw inspiration and direction from the many high-quality examples of learning collaboratives that have been conducted in behavioral health and the wealth of available educational and training resources. Best practices, as drawn from the literature review and expert interviews, focus on the importance of maximizing interpersonal interaction (in person and virtually), carefully selecting participating organizations and their change teams, conducting pre-work, and assisting teams in implementing interventions and measuring change. Adopting a health equity lens that shapes every aspect of a collaborative is considered essential.
Learning collaboratives are viewed as a heavy lift, involving significant costs, time, and effort. 66 In an article on engaging behavioral health providers in learning collaboratives, Jensen-Doss et al. suggested that we must "…reconcile the fact that less intensive training methods, such as one-time workshops, are feasible for participants, yet generally ineffective for creating sustained practice change, whereas more intensive, more effective training methods are challenging for many providers to complete." 77(p.288-289) The role of a technical assistance provider is to select an effective improvement and implementation strategy, as well as the agencies and individuals that can participate effectively in that change effort. At the same time, leaving behind other provider agencies because they have less resources and capacity is not justifiable. Such agencies are more likely serving diverse communities that already suffer from serious health inequities. While creative models and numerous resources inform the use of learning collaboratives and other comprehensive strategies to improve practice and health, more intensive advocacy is crucial to ensuring that our communities and the providers that serve them have the resources and supports necessary to be part of these solutions.