Administration and Policy in Mental Health and Mental Health Services Research

, Volume 35, Issue 1, pp 66–72

Building Bridges to Evidence-based Practice: The MacArthur Foundation Child System and Treatment Enhancement Projects (Child STEPs)

Authors

    • Family Services Research CenterMedical University of South Carolina
  • Kelly Kelleher
    • Columbus Children’s Hospital and The Ohio State University
  • John R. Weisz
    • Judge Baker Children’s Center
    • Harvard Medical School
  • The Research Network on Youth Mental Health
Brief Communication

DOI: 10.1007/s10488-007-0160-9

Cite this article as:
Schoenwald, S.K., Kelleher, K., Weisz, J.R. et al. Adm Policy Ment Health (2008) 35: 66. doi:10.1007/s10488-007-0160-9

Abstract

The papers in this special issue describe research undertaken by the MacArthur Foundation-funded Research Network on Youth Mental Health. The project is designed to understand the challenges of implementing evidence-based treatments in community-based mental health practices. This Introduction and the following articles describe the impetus and conceptual framework underlying one cluster of the Network’s activity—i.e., the Clinic Systems Project (CSP). The CSP studies examined the organizational and service system environments in a large national sample of community mental health and affiliated organizations that provide services to children. The main goal is to identify leverage points for, and barriers to, the adoption and implementation of evidence-based practices for children.

Keywords

Evidence based treatmentsEvidence based practiceChildren’s mental health

The Child System and Treatment Enhancement Projects (Child STEPs) initiative was launched in 2003 by the Research Network on Youth Mental Health,1 funded by the John T. and Catherine D. MacArthur Foundation, to help bridge the science-practice gap in children’s mental health services. Science-practice gaps have a long history in this nation’s health (including mental health), education, and welfare enterprises. In 1963, Senator Hubert Humphrey issued the following call to action in the American Psychologist: “We need people to build bridges from research to community programs. The bridges must lead from scientific symposia to the halls of Congress, to Federal office buildings, state legislatures, city halls, school boards, chambers of commerce, trade unions, service clubs, PTAs, churches and temples, neighborhoods, street corners, and every other arena of opinion and action “ (Humphrey 1963, p. 291).

In the subsequent four decades, the existence, consequences, and ostensible causes of the gap between scientifically tested innovations and dominant practices, along with proposed remedies (i.e. the ‘bridges’), have been periodically reviewed by Congress, the Office of the President, the Surgeon General, federal research and service funding agencies, and independent reviewers (see, e.g., Institute of Medicine 2001; Melton and Pagliocca 1992; President’s New Freedom Commission 2003; National Institute of Mental Health 1971, 2001; Racine 2006; Stoltz 1981; U.S. Public Health Service 2000).

In children’s mental health, concern about the science-practice gap and possible strategies to bridge it intensified in the last decade. Among factors contributing to that intensification was the publication of evidence supporting the effectiveness of child psychotherapies in research (Weisz and Weiss 1993; Weisz et al. 1995c) and illuminating the gap between outcomes of child treatments as implemented in research versus practice settings (Weisz et al. 1995a, b). These findings emerged as federal and state agencies were embarking upon the proactive dissemination of child mental health service system reforms. The objective of the reform efforts was to stem the rising tide of restrictive placements of children with serious mental health problems outside the home by making child and family-centered, community-based services more accessible to these children and their families. The principles of the systems reform had been articulated (Stroul and Friedman 1994), and policies, funds, and service coordination strategies emanating from these principles were being examined (Bickman 1996; Bickman et al. 1997). Federal funding for broader dissemination of the system of care to a greater number of states was announced prior to the completion of those studies, which ultimately found that reforms improved service access and satisfaction, and increased costs, but did not improve youth outcomes. Nevertheless, federal expenditures to support systems of care have been significant.

Indeed, the press to attenuate human suffering often translates into a push to speed the dissemination of untested or inadequately tested innovations using inadequately tested implementation strategies (e.g., Adams 1994; Backer et al. 1995; Brown 2000; Stolz 1981; Weiss 1972). In contrast, the conservative nature of sound scientific inquiry seems to risk stalling indefinitely the building of bridges. The science of bridge building is underdeveloped, in part because theory and research pertinent to bridging practice–research gaps is scattered across many fields; poorly indexed; usually considered from the perspective of a single discipline (e.g., organizational, versus economic, versus psychological); and plagued by methodological limitations (Fixsen et al. 2005; Grimshaw et al. 2001; Grol and Grimshaw 1999; Schoenwald and Hoagwood 2001; Stirman et al. 2004). Scientifically tested ways to bridge practice–research gaps in mental health services are thus difficult to catalogue with confidence, while theory, principles, and case study examples are easily accessible and often compelling.

Recently, models for speeding the scientific progression from treatment effectiveness to dissemination in children’s mental health have been proposed (Hoagwood et al. 2002; Weisz 2000), studies of the transport and implementation of effective intervention models for youth populations are underway (see e.g., Chamberlain 2003; Schoenwald in press), and federal funding agencies have increasingly focused on catalyzing gap-bridging research (National Institutes on Drug Abuse 2004; NIMH 2006). The MacArthur Foundation Child STEPs initiative is an example of such research.

The MacArthur Foundation Child STEPs Initiative

With support from the MacArthur Foundation, the Child STEPs initiative focused on generating empirical evidence about service systems, provider organizations, treatments, and children and families needed to develop and test gap-bridging strategies for children with the problems that account for a large proportion of outpatient child mental health treatment referrals, namely anxiety, depression, conduct problems, and their co-occurrence. The initiative aimed to increase the speed of the progression from treatment efficacy to dissemination, without either compromising the safety of children and families or “poisoning the waters” (Schoenwald and Hoagwood 2001) of practice and policy for the adoption of other effective treatments by prematurely going to scale. Child STEPs thus set out to simultaneously develop an empirically informed snapshot of the nation’s infrastructure for children’s mental health services and conduct a randomized trial of alternate approaches to implementing empirically supported treatments (ESTs) in practice contexts reflective of that infrastructure, so that the viability of broader implementation, and strategies to address empirically identified service system, organizational, and work force constraints on such implementation would already be established when the randomized trial results identify the most effective treatment approaches.

There are two major components from this phase of Child STEPs. The Clinic Systems Project (CSP) and the Clinic Treatment Project (CTP). The CSP is the focus of this special issue. The CSP included: (1) the Director’s Survey, a structured interview survey of 200 directors of mental health organizations serving children that focused on their governance structures, financing and reimbursement structures, (2) the Family Advocacy Survey, a structured interview survey of consumer advocacy organizations in the same communities as the target organization; and, (3) the Organizational Social Context (OSC) assessment, completed on-site by clinicians in these mental health organizations.

The CTP is a multi-site randomized treatment and effectiveness trial executed in community mental health service organizations. The CTP tests the effectiveness and implementation of two different ways clinicians might deploy empirically-supported treatment procedures, following standard manualized protocols, or using the modules from these manuals more flexibly, for highly prevalent problems of school-aged children referred for outpatient treatment. The issues addressed by the CTP, project sites, and research protocol have been described elsewhere (Martin et al. 2007). In addition, a qualitative study of examines the experiences of participants in the CTP with clinical and research procedures.

Collectively, the Child STEPs studies were designed to generate findings to guide the design of treatment, clinical implementation, organizational, and service system strategies to support the successful larger-scale implementation of effective outpatient treatments for common childhood disorders.

Conceptual Framework Guiding Child STEPs

The conceptual framework guiding Child STEPs is depicted in Fig. 1. The framework is intended to convey the dynamic patterns of influence that arise within and among key elements of the children’s mental health system. These key elements include, at the macro level, the government agencies with mandated responsibility for the oversight and financing of services; the provider organizations that are subsumed within or contracted by these state systems and to provide mental health services; the staff who work within these organizations; the specific treatments and services that are delivered; and the children and families themselves. The framework was developed on the basis of a synthesis of theory and research on treatment effectiveness, organizational behavior, health care services improvement, innovation diffusion, technology transfer, and the transport of ESTs to community based mental health settings. The model depicts “the emerging concept that broad-based implementation has a systemic nature and underpinnings, with facilitators and barriers at the level of policy and regulations, the level of the organization, the level at which service provision or treatment occurs, and the levels of the consumer and family member” (Stuart et al. 2002, p. 328).
https://static-content.springer.com/image/art%3A10.1007%2Fs10488-007-0160-9/MediaObjects/10488_2007_160_Fig1_HTML.gif
Fig. 1

Conceptual model for the MacArthur Research Network on Youth Mental Health Child STEPs initiative on evidence based practice in clinics and systems

The Clinic Systems Project (CSP) Papers

The first paper in the special issue (Schoenwald, Chapman, Kelleher, Hoagwood, Landsverk, Stevens, Glisson, & Rolls-Reutz) reports the results of the Director’s Survey, conducted via structured interviews of the directors of 200 mental health service organizations across the country treating children. The survey was designed to provide a contemporary snapshot of the nation’s infrastructure for community-based children’s mental health services, and thus focused on the governance, financing, staffing, services, and implementation practices of these organizations. This snapshot is pertinent to the design and testing of strategies to facilitate the adoption and implementation of empirically supported treatments (ESTs) for children, because, to be successful, such strategies will likely have to both accommodate and change features of that infrastructure (i.e. provider organizations and the service systems within which they are embedded.)

Survey results indicate the majority of provider organizations serving children are private, staffed by licensed professionals, and experienced in the adoption and implementation of new treatments and services, and that Medicaid is an important source of service funding. The survey also asked directors about factors they considered important to the successful implementation of new treatments and services, and among the findings are that infrastructure support was important in public but not private organizations. Results of analyses examining relations between infrastructure characteristics and implementation of new treatments and services indicated for profit status was associated with greater likelihood of such implementation.

The second paper in the issue reports the results of a companion survey of consumer advocacy organizations operating in the communities represented in the Director Survey sample. Under the leadership of Network members Kimberly Hoagwood and Evelyn Green, this study was undertaken with supplemental funding from the Robert Wood Johnson Foundation after data collected from the Director’s Survey suggested that directors were aware of mental health consumer advocacy groups operating in their midst, and some evidence of consumer involvement in the activities of the mental health organization itself. Survey results indicated the majority of consumer advocacy organizations were small local organizations funded primarily by private donations with affiliations to national consumer advocacy groups. The local organizations had varying degrees of formal and informal linkages to community-based service provider organizations. Although a range of services were provided by the advocacy organizations, the primary service they provided was information and provision of referrals to families. Although issues traditionally important to grass-roots advocacy for children and families such as increased public awareness and family involvement were also identified as important in this survey, availability of accurate screening and assessment, evidence-based practices, and cultural responsiveness were identified as important by over half of the sample.

The third and fourth papers in this issue focus on understanding the social context of mental health organizations serving children and families in terms of constructs associated either with the adoption, the implementation, or sustainability of new technologies in other industries. Organizational culture, climate, and structure, and worker attitudes are among constructs linked with organizational openness to, and success with, innovation in other industries and service sectors. In the Organizational Social Context (OSC) component of the CSP, one hundred of the provider organizations in the Director’s Survey sample completed a battery of validated measures of these constructs in on-site meetings. The first paper by Glisson and colleagues (Glisson, Landsverk, Schoenwald, Kelleher, Hoagwood, Mayberg, & Green) describes the distinctive profiles of organizational culture and climate that characterized this large nationwide sample of child-serving mental health organizations. Clinicians’ work attitudes were found to differ significantly in the organizations with the best and worst culture and climate profiles providing the first large multi-state assessment of such factors in child human service agencies.

The next paper (Glisson, Schoenwald, Kelleher, Landsverk, Hoagwood, Mayberg, & Green) reports the results of analyses examining the relations between these organizational social context profiles, workforce turnover, and the sustainability of new treatments and services as reported in the CSP Director’s survey (Schoenwald et al. this issue). Negative aspects of organizational climate were associated with reported workforce turnover, while positive aspects of organizational culture were associated with the sustained implementation of new treatments and services. Contrary to the hypotheses suggested by the CSP model, external environmental factors and internal structures were of limited importance in sustainability of new treatments and services.

The final paper features the use of data to inform clinical and organizational decision-making in children’s mental health services through a computerized management information system. This paper describes a reporting system designed to facilitate the clinical and administrative use of evidence-based practices. The system was originally developed by Chorpita and colleagues in the context of a statewide initiative in Hawaii to implement evidence-based assessment and treatment practices and has subsequently been tailored for use with other systems, including those participating in the CTP component of Child STEPs. This paper describes the rationale, design, and use of effectively designed information management systems to simultaneously support treatment progress and organizational accountability for implementation and outcomes.

The collection of papers in this special issue provides a national perspective on the infrastructure, organizational social context, and implementation practices of community-based children’s mental health services, and considers them relative to the demand characteristics of evidence-based practice implementation. Taken together, the papers provide some empirical grounds with which to build bridges that will link our research on treatments with community practice in children’s mental health. First, the infrastructure for community-based children’s mental health services appears to be largely publicly (combination of Medicaid and other government entity) funded, but privately operated. Public systems are not, therefore, providing the majority of services for children. The prevalence of private provider organizations and increased odds of using new treatments and services associated with organizational for-profit status suggest private providers are innovating, and that collaboration between public payers and private providers may be needed to establish financial and operational conditions for the adoption and implementation of an evidence-based treatment. Second, provider organizations are accustomed to clinical program change, reportedly implementing a new treatment or service on average once every one to two years. Thus, the prospect of change in and of itself may not be a barrier to the implementation of evidence-based practices; the nature of the change required, however, may indeed be an issue. There appears, for example, to be at least some compatibility between the training and clinical supervision practices characterizing mental health organizations and those associated with some evidence-based treatments. On the other hand, mismatches may characterize indicators used to assess clinical productivity, the demand characteristics of some treatment and service delivery models, and successful treatment implementation. Thus, alignment of productivity indicators with evidence-based practice contours may be needed to facilitate successful implementation.

Family advocacy and support organizations, while characterized by fiscal instability, stimulate public awareness and family involvement in children’s mental health services, and identify as potential advocacy issues the use of adequate screening and assessment, evidence-based treatments, and culturally responsive practices. Taken together with the perceived importance to service effectiveness of well-trained clinicians and high quality relationships, the interests of family advocacy and support groups overlap considerably with those of service provider organizations, treatment and services researchers, and research funding agencies. As noted elsewhere (NIMH 2006) leveraging the alignment of interests across these consumer, provider, research, and funding agencies will be important to the implementation of evidence-based practices.

The mental health organizations providing services for children vary considerably with respect to culture and climate, and workforce attitudes, factors that have been associated in other research with organizational innovation, workforce turnover, and service quality. There are several implications of these findings for the development of strategies to effect the larger scale dissemination of evidence-based practices. First, dissemination strategies might target organizations with profiles indicating positive climate and culture as early adopters of a new, evidence-based practice (Rogers 1995). In addition, in organizations with climate and culture profiles less favorable to the sustainability of the clinical work force and of new treatments, organizational interventions designed to change climate and culture may be needed to facilitate adoption and implementation. Finally, technological innovations that produce relevant, non-redundant, interpretable information to inform clinical and administrative practice decisions in real time is essential to support the bridge between research and practice.

Bridge-building relies on a strong foundation and appropriate materials and design. To think that bridging the gap between research and practice for child mental health would be any different is foolhardy. The information provided from these diverse studies provides a strong foundation about the current delivery of mental health services to children and adolescents along with some suggestions for tools to start building that bridge. Multi-phasic trials are the next logical step to accomplish our goals.

Footnotes
1

The Research Network on Youth Mental Health is a collaborative team funded by the John D. and Catherine T. MacArthur Foundation. Network Members at the time this work was performed included: John R. Weisz, Ph.D. (Network Director), Bruce Chorpita, Ph.D., Robert Gibbons, Ph.D., Charles Glisson, Ph.D., Evelyn Polk Green, M.A., Kimberly Eaton Hoagwood, Ph.D., Peter S. Jensen, M.D., Kelly Kelleher, M.D., John Landsverk, Ph.D., Stephen Mayberg, Ph.D., Jeanne Miranda, Ph.D., Lawrence Palinkas, Ph.D., Sonja K. Schoenwald, Ph.D.

 

Acknowledgement

Preparation of this manuscript was supported by the John D. and Catherine T. MacArthur Foundation and grant DA018107 from the National Institute on Drug Abuse.

Copyright information

© Springer Science+Business Media, LLC 2007