Abstract
People engaged in efforts to improve services to emerging adults with serious mental health challenges have reached the conclusion that service change at the program or agency level is not sustainable without related changes at the systems or policy level. This article focuses on one set of efforts to create intentional system change at both the community and state levels. These changes were pursued by states and communities that received grants under the federally funded Healthy Transitions Initiative (HTI), with the aim of creating more effective services for emerging adults with serious mental health conditions. The article reviews the development of a measure to assess systems change efforts at the state and community levels and describes the findings that emerged when the measure was used to assess the change that occurred in the HTI sites over a period of approximately three and a half years.
Similar content being viewed by others
Over the past 5 years, the efforts of both researchers and practitioners have begun to provide insight into the best ways to design and provide services and supports for “emerging adults”—older adolescents and young adults—who experience serious mental health conditions.1 – 3 People engaged in efforts to improve services to these young people, and to other high risk populations, have increasingly reached the conclusion that service change at the program or agency level is not sustainable without related changes at the systems or policy level.4 , 5 Consequently, attention has turned to identifying and testing approaches for driving intentional change at the system level(s).6 – 8 This article focuses on one set of efforts to create intentional system changes at both the community and state levels. These changes were pursued by states and communities that received grants under the federally funded Healthy Transitions Initiative (HTI), with the aim of creating more effective service systems for emerging adults with serious mental health conditions. This article reports on the development of a measure to assess systems-change efforts at the state and community levels and describes the findings that emerged when the measure was used to assess the change that occurred in the HTI sites over a period of approximately three and a half years.
One of the most often-cited definitions states that a system is a collection of parts that interact together and function as a whole.9 This means that systems can come in many different shapes and sizes, and sometimes it is difficult to determine exactly what is meant when a “system” is referenced. For example, a family, a neighborhood, or a community can be considered a system. Similarly, an organization, a set of service delivery programs, a coalition of agencies, or the complex set of federal funding agencies can be considered a system. It is thus not surprising that the first step in discussions of systems typically involves “bounding” or identifying the parameters of the system to be examined. Foster-Fishman et al. offer a more nuanced definition of a system: “the set of actors, activities and settings that are directly or indirectly perceived to have influence in or be affected by a given problem situation.”10 (p.198)
This article focuses on systems and systems change at two levels: the community and the state. The community-level system includes the individuals and programs in a local community that work together with the goal of identifying, engaging, and providing comprehensive, integrated services to emerging adults with serious mental health conditions. The state-level system includes individuals and programs that have some kind of state-level authority for planning, funding, or guiding community-based services for emerging adults. System change is defined as an “intentional process designed to alter the status quo by shifting and realigning the form and function of a targeted system.”10 (p.197) Systems change is seen as a crucial ingredient in efforts to improve services for emerging adults since, in the absence of adequate funding and a supportive policy environment, it is difficult to maintain high-quality programs and interventions with fidelity.11
A clear theory describing the process of change at the systems level (as opposed to the individual service delivery level) has yet to be articulated and accepted. Consequently, current understanding regarding the best way to create planned change to a system of services is limited, and most planners and administrators employ a “muddling through” approach,12 in which members of a system take short-term steps that seem to be leading in general direction of desired change. Without a theory of change, however, it is difficult to create longer-range strategic plans to guide and coordinate efforts across the system, and so actors are left to muddle through and hope for improved outcomes.
Foster-Fishman et al.10 offer a framework for changing organizational and community systems. The framework is grounded in systems thinking and change literature. They propose to identify system parts, understand the interrelationships among those system parts, and use that understanding to identify leverage points that can be employed to cause the system to change. This framework has four steps which are labeled as the “essential components of transformative systems change.”(p.202) These four components are (1) bound the system, (2) identify fundamental system parts, (3) assess system interactions, and (4) identify levers for change. Bounding the system requires two substeps. The first of these is to clearly define the problem and second is to identify the actors, organizations, and multiple levels of systems that are related to this problem. Understanding the system parts requires the actors to explore system norms, resources, regulations, and operations. This is followed by a process of understanding the interactions and interdependencies among system components and the way the system self regulates. The final step provides the greatest contribution to understanding how to change systems. This step involves identifying levers that can be used to change the system and includes identifying system parts that could be the target of change and identifying the interaction patterns that could be leveraged for change. Although the specific mechanisms for making change happen at the system level are not explicated, this framework offers a format for identifying the leverage points that might be the focus of intentional intervention.
Emshoff et al.13 combine ecological and systems theories to examine change created by community collaboratives in the health care system. Community collaborations in health care are seen as a way to address complex challenges that require interrelated solutions. These challenges include the need to make maximum use of resources, the need to reduce duplication of services across systems, and the need to include consumer voice in decision making. Emshoff et al. posit that there are three elements that can be a target for system change: decision making, financing, and collaborative and accessible service delivery. Decision making refers to how the community collaborative is governed and how decisions are made about which services will be provided. Financing involves how resources are obtained and allocated across the work of the collaborative. Collaborative and accessible services delivery is concerned with what types of services will be developed and offered and how they will be delivered. Reducing the focus of system change to these three targets may be premature as far as evidence goes, but this strategy does offer a way for community collaboratives and local systems to get a toe-hold on how to manage the change process with less muddling.
Review of the existing literature on systems and systems change also points up the lack of standardized instruments or performance indicators for measuring system change.14 , 15 One of the measurement approaches that has been developed is associated with the evaluation of the Children’s Mental Health Initiative, funded under the federal Comprehensive Community Mental Health Services for Children and Their Families Program. The program funds states and communities seeking to improve the extent to which they have developed systems of care—i.e., a coordinated and comprehensive array of effective services and supports for children with serious emotional and behavioral disorders and their families. The measure, called the System of Care Assessment (SOCA),15 is organized into two domains—infrastructure and service delivery. Within each domain, general system components are assessed. For example, within the infrastructure domain, governance, management and operations, and service array and quality monitoring are examined. This extensive framework is organized into a matrix of components and principles, which in turn are further expanded into indicators that serve as the basis for data collection. Data collection is done through a 3–5-day site visit by trained site visitors. Site visitors conduct interviews and review documents to rate the extent to which the system has achieved each indicator. Individual systems of care are assessed at least three times during a 6-year funding period.15 Although expensive to administer, this approach to data collection results in finely grained information about each domain, as well as an indication of change over time. A related approach to assessment is found in the Case Studies of System Implementation (CSSI) developed by Hodges et al.16 This approach uses a multisite, embedded case study design to assess system change.
At the quantitative end of the measurement spectrum, social network analysis can be used to document interagency networks and show how information, resources, and clients are flowing among organizations in a network.17 , 18 Social network analysis requires that lengthy interviews be completed by everyone in the identified system so that specific analytic approaches may be used. Further, social network analysis focuses on coordination and collaboration among organizations and does not examine other kinds of systems activities that might be precursors or outcomes of system change.
A less labor-intensive approach involves collecting data via a web-based survey or questionnaire. An example of this is the Community Supports for Wraparound Inventory (CSWI), an assessment tool that has its origins in qualitative research on the implementation context of wraparound. Wraparound is the most frequently implemented approach to realizing the system-of-care principles at the service delivery level. Research on the systems context of wraparound resulted in a clearly defined set of themes related to systems change and items to measure those themes.19 Tools such as the CSWI are intended to provide enhanced focus on key themes or elements identified by stakeholders and to provide clear information to help drive system change. One of the limitations of the CSWI is that it depends on local community leaders to identify who should respond to the survey. The process of “bounding the system” is critical to all measurement efforts, and it is unknown whether local stakeholders with limited guidance are able to perform this task adequately. Another concern involves the dependence on assessments made by community members who may have expertise in only some areas of the system. This concern can be leveled at all attempts to measure system change, although those approaches that employ more than one approach to measurement may have a slight advantage.
System change, as a focus for research, presents many challenges and much complexity. Although efforts have been made to articulate a framework for how systems change happens, studies are needed to test these frameworks and to develop a workable theory of change. Tools for assessing systems change can provide information that is valuable in moving this work forward; however, the few measurement tools that have been developed to date tend to be time consuming and expensive to use. A challenge now is to develop, refine, and test an efficient way to gather reliable, useful data about key aspects of system change.
This study reports on the development and pilot testing of two measures—one at the state level and one at the local community level—that are intended to assess the extent to which these systems have developed the capacity to provide comprehensive, coordinated services and supports to emerging adults with serious mental health conditions. The aims were (1) to create reliable assessments that could be administered via a web survey of stakeholders at the state and local levels and (2) to use the assessments to examine systems change in states and communities in which efforts were underway to improve the system response for emerging adults with serious mental health conditions.
Methods
This article describes the development of assessments of the extent to which states and communities have developed the capacity to provide comprehensive, coordinated services and supports for emerging adults with serious mental health conditions. The article also describes a pilot test of these assessments, which were used to examine system change accomplished through the Healthy Transition Initiative (HTI) funded by the US Substance Abuse and Mental Health Administration (SAMHSA) for 5 years, beginning in 2009. Each of the seven grantee states implemented HTI services in at least one local community, with three states implementing services in two communities. In addition to identifying and implementing an evidence-informed model for service delivery—and engaging emerging adults in services—other goals of the grant included bringing together relevant stakeholders at both community and state levels, identifying system level issues that needed to be addressed and mounting an action plan to effect change in some aspect of the system, such as policies, structure, procedures, or funding mechanisms.
The data for this study were collected using the Community Support for Transition Inventory (CSTI) and the State Support for Transition Inventory (SSTI), two web-based tools developed by the Research and Training Center on Pathways to Positive Futures (Pathways RTC) at Portland State University in Portland, Oregon, and made available for use by the HTI sites. The CSTI was designed to serve as a guide to help communities understand both what they are aiming for—sustainable capacity to provide effective, comprehensive support for young people with serious mental health conditions—and how much progress they have made in achieving that goal. The SSTI recognizes the important role that state-level infrastructure and polices can play in helping—or hindering—local efforts to make these fundamental changes. The SSTI is an assessment that is designed to give stakeholders reliable, objective feedback about the extent to which the state has developed the capacity to support local efforts. Communities and states can use the information from both CSTI and SSTI as an input for strategic planning. Repeated use—at intervals of 2 years or so—allows each entity to objectively assess what it has accomplished and what yet needs to be done.
Development of CSTI and SSTI
The CSTI is an adaptation of a measure called the Community Supports for Wraparound Inventory,19 which assesses the extent to which community partners have come together to provide comprehensive, community-based care for children with serious emotional or behavioral disorders and their families. The CSWI emerged from research conducted at Portland State University, in collaboration with experts from the National Wraparound Initiative, an organization that works to provide stakeholders with resources and guidance that facilitate high quality and consistent wraparound implementation. A study of the CSWI has provided evidence of the measure’s reliability and validity.19 , 20
The children and families who participate in Wraparound typically receive services and supports from multiple agencies and systems, and in most cases, the children are at high risk of being placed in an institutional setting such as a hospital or residential treatment center. Wraparound brings the family and providers together to create a single, collaborative, and comprehensive care plan that is designed to ensure that the child can remain in the community and thrive. Efforts to serve emerging adults with serious mental health conditions are undertaken in a system environment that is somewhat different from the environment for Wraparound programs, and the needs of youth and young adults are different from those of children. Nevertheless, the basic underlying challenge is the same: to bring diverse systems and providers together to provide comprehensive, coordinated, community-based care focused on improving functional outcomes and quality of life. Thus, the CSWI was seen as an appropriate starting point for efforts to develop an assessment of community support for comprehensive transition initiatives.
Adaptation of the CSWI began with a review of items by the research team at Portland State University. This was followed by several rounds of review and further adaptation based on feedback solicited from individual stakeholders knowledgeable about transition services. Feedback was sought from stakeholders with high levels of experience and expertise, including young people and families, providers, administrators, researchers, and state-level policy makers. For the first round of feedback, nine stakeholders participated in reviewing and editing each of the items proposed for the new measures. They also provided feedback regarding the importance of each item and the extent to which the items associated with a particular theme adequately “covered” that dimension of system support. The research team and the stakeholders worked together to review the feedback and create new versions of the assessments. For the next two rounds of feedback, an additional nine stakeholders (five in the second round and four in the third) completed the online assessments (using their own communities/states as the focus of assessment) and provided feedback about the items and themes.
The finalized version of the CSTI maintains seven themes in parallel to those included in the CSWI but adds a further theme focused on state-level support for local efforts. The assessment thus provides scores on eight themes measured by 45 items. The eight themes, and representative items, are shown in Table 1. Participants respond to each item on a 5-point scale from “least developed” to “fully developed,” with corresponding ratings from zero to four (higher scores indicate a more fully functioning aspect of the system). Each item includes a full description for least developed and fully developed (examples are shown in Table 1). The other points on the scale are described as “some development,” “midway,” and “almost there.” The SSTI parallels the CSTI and includes 26 items organized around six of the eight themes. The SSTI does not include items related to “practice quality and support” or “state support.” Responses are provided on the same 5-point scale.
In addition to the CSTI and the SSTI, the research team developed a youth/young adult module that requests feedback from young people who have personal knowledge about the services and supports they received while participating in a transition program. Findings from work related to this module will be reported elsewhere.
Data collection
In each HTI state, two distinct sets of individuals were recruited to respond to the web-based surveys. The CSTI is intended for individuals at the community level who are involved with system-wide efforts to provide “transition” services to emerging adults with mental health conditions and related needs. While this may include staff from transition-specific programs and staff from agencies or programs that refer to or receive referrals from transition programs, respondents are primarily individuals who serve on community-level advisory groups or governing bodies that are concerned with transition issues. Young people, family members and other allied adults who serve on advisory boards or who are significantly involved with planning and evaluating services for emerging adults are also asked to respond to the CSTI. HTI grantees are expected to include young people and family members on these decision-making bodies that oversee systems-change work at both the community and state levels.
The SSTI is administered to individuals at the state level who are involved with planning and funding services to emerging adults with mental health needs. This list usually includes administrators and/or staff from state divisions of mental health, child welfare, education, vocational rehabilitation, and other state divisions that have responsibility for some aspects of services to young people. The SSTI is also completed by youth and young adult and adult allies who are active in promoting, planning, or overseeing services at the state level and as well as other members of state-level advisory groups or governing bodies.
The data from the CSTI and SSTI were collected from HTI stakeholders at the state (seven states) and community (ten communities) levels at two points in time. Time 1 (T1) occurred when the HTI grants were just getting underway and Time 2 (T2) occurred in the fourth year of project implementation. In the first step for the CSTI, the community selects a local coordinator to work with the research team from Pathways RTC. The local coordinator has two main responsibilities: to work with the research team to compile a list of potential respondents and to work in the community to make sure that the identified respondents understand the purpose of the CSTI, the value of the data to the community, and the importance of completing the CSTI. The local coordinator is given information about the types of people that are typically invited to respond to the CSTI. The local coordinator is also provided with a list of sample items and asked to include on the list only people who they believe could respond to most of these. A member of the research staff reviews the list provided by the local coordinator to answer any questions about who should be included. The goal of this work is to ensure a good response rate, so that the community can have confidence that the findings from the CSTI are indeed an accurate representation of the perceptions of stakeholders who are centrally involved in systems-change efforts. The size of the list of potential respondents depends on the size of the community and, to some extent, on how much development has already taken place. On average, the initial lists for the HTI sites were between 15 and 25 individuals.
The research team then creates an online version of the CSTI that is specific to the community (i.e., the survey references the transition project using the state/community name and the name by which respondents know their state/local HTI project) and sends an e-mail invitation to each potential respondent. Potential respondents are given about 3 weeks to complete the CSTI and are sent weekly e-mail reminders. People can choose to “decline” the survey, after which they do not receive further reminders. Individuals who do not decline continue to receive e-mails and follow-up calls asking them to respond or decline. The research team and the local coordinator monitor the response rate—75% is the target. Response rates for this study ranged between 31 and 97% for the CSTI.
The data collection for the SSTI follows a similar pattern. A state-level coordinator is appointed who knows the policy makers at the state level and can identify potential responders and follow-up with them to assure an adequate response rate. Roughly 20–25 potential respondents were identified at the state level, and response rate between 47 and 77% was achieved. One state was unsuccessful in engaging state-level stakeholders at T2 and thus had data only at T1.
Data analyses
This study utilized multilevel models to examine the effects of individual-level variables and the site of data collection, using R, version 3.0.2.21 Since these data have a hierarchical structure, with respondents nested within sites, multilevel techniques were used.22 , 23 The analysis began by fitting null models to examine between-group effects for each dependent variable (entire scale scores and individual theme scores). Covariates—race and role in the project—and the predictor of interest, time, were added to these null models.
It was only possible to match about 30% of the respondents from T1 to T2. This was not only due to turnover and individuals changing roles within the sites but also due to inconsistencies in the way respondents were identified at T1 versus T2 (i.e., by name or role) and the fact that some people did not use their individualized links that were provided by the survey platform. As a result, it was not possible to measure within-individual change from T1 to T2. Instead, multilevel regressions were conducted to examine differences in mean scores from T1 to T2. Separate multilevel models were constructed for the state-level data (SSTI data) and community-level data (CSTI data).
Results
After the data set was cleaned, histograms for each item were examined, and skewness and kurtosis statistics were computed, and there were no worrisome deviations from the normal distribution. To handle missing data, multiple imputations were performed on both the state and community datasets using the “Amelia” package24 on default settings. Twenty imputed datasets were generated and averaged together for the final dataset used for analyses. Due to extensive missing data, two respondents were dropped from the imputed dataset, leaving a final sample size of 260 for the community data. No respondents were dropped from the imputed state dataset, leaving its final sample size at 170. See Table 2 for the sample sizes, broken down by covariates, at T1 and T2 for the community data, and Table 3 for the state data. Responses from emerging adult consumers of services were included in the “other” category due to extremely small numbers. There were only two young people represented in the CSTI data (both time 2) and none in the SSTI data.
Reliability and confirmatory factor analysis
To determine reliability, Cronbach’s alpha for the overall survey and individual themes was examined for both time points. Cronbach’s alpha for the CSTI as a whole was .98 at both time points and for the individual themes ranged between .81 and .94 at T1, and .88 and .94 at T2. For the state data, Cronbach’s alpha was .95 at T1, and .92 at T2 for the SSTI as a whole and, for the individual themes, ranged from .83 to .88 at T1, and .63 to .85 at T2.
Confirmatory factor analyses on the CSTI and SSTI were conducted using the Lavaan package.25 All items were specified to load freely on their intended factor. The metric of each factor was set by fixing the factor loading of the first indicator to one. Model fit was assessed with a combination of goodness-of-fit indices: chi-square, comparative fit index (CFI), Tucker-Lewis Index (TLI), root mean squared error of approximation (RMSEA), and the standardized root mean square residual (SRMR), following the recommendations of researchers.26 For the model to be deemed an acceptable fit for the data, the chi-square statistic should be nonsignificant, indicating that scores from the model do not significantly differ from those observed. However, due to its sensitivity to large sample sizes, the chi-square statistic is often overly conservative, as it nearly always rejects sufficient models.27 As such, attention was focused on the other fit statistics. Generally, values greater than .95 for the CFI and TLI indicate good model fit, whereas values between .90 and .95 indicate adequate fit and values under .90 indicate poor fit. For the RMSEA, a good model fit is generally reflected by a value of .06 or lower, while values between .06 and .08 are fair, .08 to .10 are adequate, and values above .10 are poor. For the SRMR, the conventional cutoff criterion is less than .08 for a good-fitting model.28 Full information maximum likelihood was used for estimating model parameters. Modification indices (MIs) were inspected and parameters were added to the model only when they were deemed consistent with theory. MIs indicate the estimated amount of chi-square reduction achieved if there is a modification to the parameter in question in the model.
The community factor structure retained the eight factors as originally defined, with all eight factors loading onto a higher-order factor indicating the overall score of the CSTI. Similarly, the state factor structure retained the six factors as originally defined, again with all six factors loading onto a higher-order factor indicating the overall score of the SSTI. See Table 4 for the fit indices of both models. Factor scores for themes and the overall scales were determined for the imputed dataset and used for the next set of analyses.
Multilevel analyses
Null models were computed for community and state data. Next, models were computed using race/ethnicity, current role, intensive service status (i.e., whether or not the respondent or any member of his/her immediate family had received intensive mental health services) and, of most interest, time. Covariates were entered into the models before time. These models were then expanded on by examining two-way interactions between time and each of the covariates. Intercepts and slopes for time were set as random effects, following the recommendations of Barr et al.29 Means at T1 and T2 were calculated for both the CSTI and SSTI and for each of their subscales/themes using factor scores and imputed data.
Table 5 shows mean scores and slopes at T1 and T2 for the overall CSTI and for each of its themes, for all of the local communities taken as a group and for the communities individually. Considering all of the communities as a group, neither the overall CSTI nor any themes were rated significantly higher at T2 than T1 (i.e., slopes significantly different from 0 in a positive direction), though there is a trend toward significance for fiscal policies and sustainability and accountability. The interactions between time and the various covariates (including race and role in the project, not shown in the table) were nonsignificant for the most part. One interesting exception is that community members (stakeholders who were neither service providers nor employed by the project) rated community partnership and collaborative action significantly lower at T2 than T1. Scores on the CSTI and its themes did change significantly for some of the individual communities. For example, community B started out at T1 with means on the CSTI and each of the subscales significantly below the respective grand means. By T2, community B’s theme means still lagged the grand means; however, the increase between T1 and T2 was positive and significant. Community E1 (one of two communities in state E) had means for the CSTI and each of its themes that were significantly above the respective grand means, and it also showed positive and significant increases for the CSTI as a whole and seven of the eight themes. Four of the ten communities had significant T1 to T2 increases on the CSTI as a whole, as well as significant increases on at least six of the themes (communities B, E1, and G) or significant or trend-level increases on five themes (community C1). Among the remaining communities, two had trend-level increases in CSTI score and several significant increases in theme scores (communities C2 and E2), and two other communities (communities A and F2) started out at T1 with CSTI and theme scores generally higher than the grand mean and maintained those scores at T2, except for one theme at each site, which saw a significant increase. The final two communities had overall CSTI scores that were generally flat from T1 to T2, with one community (community D) showing a significant decline on one theme and the other (community F1) showing significant declines on two themes.
Table 6 shows mean scores and slopes at T1 and T2 for the overall SSTI and for each of its themes, for all of the states taken as a group and for the states individually. State B did not participate in SSTI data collection at T2. Considering all of the remaining states as a group, neither the overall SSTI nor any themes were rated significantly higher at T2 than T1, though there is a trend toward significance for workforce. The interactions between time and the various covariates (not shown in the table) were all nonsignificant. One state’s scores rose significantly between T1 and T2 for the SSTI and most of its themes (state A), and two other states (states D and G) had trend-level increases in their overall SSTI scores and significant increases on two theme scores. State E had scores at T1 that were significantly higher than the grand means and maintained this advantage for three themes at T2. State F’s scores showed no significant changes, and state C’s score declined significantly for one theme.
Implications for Behavioral Health
The scale and subscale/theme reliabilities for both the CSTI and the SSTI, together with the fit indices for the final confirmatory factor analysis model, provide evidence that these assessments can generate reliable feedback on systems-change efforts in communities and states seeking to implement comprehensive, coordinated service approaches to meet the needs of emerging adults with serious mental health conditions. The engagement of stakeholders in the development of the CSTI and the SSTI speaks to the assessments’ face validity. Furthermore, the findings for the fit indices (Table 4) and the fact that the assessments were based on—and preserve the factor structure of—a similar assessment with evidence of validity also support the argument for validity of the CSTI and SSTI. However, the exploration of validity was limited in the current study, and future studies could address this issue, for example through the examination of local and state data in areas such as service access and utilization or through tracking changes in local and state policies. A further limitation stemmed from the difficulty in tracking respondents from T1 to T2. While this was partially due to actual “churn” at the sites—i.e., both turnover and people changing roles within a site—it was also partially due to inconsistencies in the way respondents were identified by the local coordinator and to the fact that some respondents did not access the survey using the individualized (hence identifiable) links provided to them. If these issues are addressed in future work with the CSTI and SSTI, it may be possible to achieve a high enough level of match to make within-subject analytical approaches possible.
Despite these limitations, the findings suggest that the CSTI and the SSTI may have potential use not only as a way of measuring current system development to support comprehensive transition projects but also as a kind of roadmap that lays out the tasks for communities and states to consider as they work through implementation from early exploration to mature implementation. The process of using the CSTI or SSTI may provide insights that help communities accomplish the steps or tasks of systems change identified by Foster-Fishman et al.10 and described in the introductory section. For example, the items from the first theme, community partnership, provide information about the key stakeholder groups—including young people and families, and representatives from a spectrum of agencies—that should be engaged in systems-change efforts. Feedback from these items thus encourages leaders of the systems-change efforts to think about how they have “bounded the system” (step 1 in the Foster-Fishman et al. framework). Similarly, feedback from individual items on the other themes can promote reflection on system resources, regulations, and operations (step 2); can point out areas in which exploration of resources and regulations needs to be undertaken (step 3); and can suggest options for systems-change “levers” (step 4), for example in the areas of workforce (theme 4) or accountability (theme 7). Additionally, the confirmation of a factor structure paralleling that of the Wraparound measure—with three of the factors also mirroring those proposed as key targets for systems change by Emshoff et al.13 and described in the introduction to this study—points to the possibility that the themes may represent areas of activity that are key targets for systems-change efforts in human services more broadly.
A limitation related to the study of the change in scores across time at the state and local levels concerns the low response rates in several of the HTI states and communities. Low responses were a particular problem at T1, when work was still getting underway and stakeholders were still being engaged. The extent to which this was a problem varied from site to site, since (as shown in the mean scores from the assessments) some sites had much higher levels of initial development than others. The low response from a few sites at T1 made it less likely that change would be detected over time and that significant findings would emerge for certain individual communities and states, as well as for the communities and states overall. It is also possible that the findings related to change over time are biased because respondents were assessing an initiative designed to promote systems change. Respondents may thus have been motivated to provide ratings showing progress over time.
The near absence of responses from young adults is also a limitation. Only three young people were nominated for the survey, of whom two responded. The low number of nominations appears to be at least partially due to low rates of participation by young adults on the planning committees or other bodies that oversee project implementation. The state and local scores on the SSTI/CCTI item referencing young adult participation in these roles were consistently the lowest among the items on the community partnership theme, making this a clear target for future work at the state and local level. Low participation may also be due to screening out of potential respondents by the local coordinator, who was asked to nominate potential participants based on the local coordinator’s perception of the potential respondents’ ability to respond to sample items on most of the themes. Participation and response rates generally for the SSTI/CSTI might be improved by recognizing that not all respondents will have expertise in all thematic areas and by encouraging the participation of respondents who are knowledgeable about some themes but not others.
In the study of the HTI grantees at the local level, the overall CSTI score for the communities as a group did not show a significant positive difference between T1 and T2. However, overall scores on the CSTI did change significantly for some of the individual communities, with four of the ten communities demonstrating significant positive change between T1 and T2, and two additional communities showing trend-level increases. No community’s score showed decline at even the trend level. A generally similar pattern was apparent for the theme scores, with three communities showing significant increases across four or more themes, and three more showing moderate to modest progress with a combination of significant and trend-level increases. Two further communities had theme scores that started out high in comparison to the respective grand means and stayed high, though without further significant increase from T1 to T2. Significant change in a negative direction (T2 being lower than T1) was relatively rare, with only one site experiencing significant negative change, and that in only two themes. The fact that most communities either had some positive results—from pronounced to modest—or maintained gains made prior to T1 suggests that it is quite possible for community stakeholders to create or sustain systems change given time and moderate resources.
While none of the themes of the CSTI showed significant positive change for the communities taken as a group, two of the themes showed a trend toward significance, fiscal policies and sustainability, and accountability. Fiscal policies and sustainability is typically an extremely challenging area for system of care projects,15 , 16 , 19 , 20 and this CSTI theme had the lowest grand mean score at T1. Considered individually, six of the ten communities made significant progress in this area between T1 and T2, while only one community experienced a significant decline. This finding sends a reasonably hopeful message about the possibility for putting comprehensive transition initiatives on a sound fiscal footing. Further investigation into the mechanisms that sites believe they employed to make changes in fiscal policies and sustainability would provide useful information to other communities and behavioral health programs as they seek to emulate this success. Similarly, in the area of accountability, seven of the individual communities had significant increases in their scores between T1 and T2, with two more showing trend-level increases and no community showing declines. This finding provides another hopeful sign with regard to the ability of community stakeholders to create system-level change.
Fewer significant changes in score between T1 and T2 were observed on the state-level assessment (SSTI). As with the CSTI, there was no significant difference between T1 and T2 on the overall SSTI or theme scores when the states were considered as a group. One state showed an increase on the overall SSTI and most of the themes, and two other states showed increases on two themes. The remaining three states had no significant score changes beyond one negative change for state C. It is quite possible that creating change across the whole state is more difficult than creating change in one community with motivated stakeholders; however, not all of the themes focused on state-wide change. For example, the first two themes, partnerships and collaborative action, focus on getting state-level stakeholders to create infrastructure for collaboration and to accomplish work together. These two themes were as unlikely to show positive change as themes focused on statewide changes, such as access to supports and services, which focuses on the statewide availability of an array of services. Furthermore, according to the CSTI data, local communities rated state support at being less than midway to being fully developed, even at T2. This finding, together with the observation that the mean SSTI scores were quite a bit lower than the CSTI mean scores, suggests that engaging state stakeholders—including state-level administrators in the various systems that serve older adolescents and young adults—is a major undertaking and can be quite challenging. Additionally, major strands of activity at the state level often involved goals that can take years to achieve, such as changing the state Medicaid plan, creating and passing new legislation, and developing memoranda of agreement among several state-level divisions. It is also possible that the methods used to make changes at the state level were less effective than those used at the community level. Either way, the findings suggest that knowledge about effective strategies for creating change in state-level systems and policies is underdeveloped and lacks a strong framework or solid theory of change.
References
Davis M, Koroloff N, Ellison M. Between adolescence and adulthood: Rehabilitation research to improve services for youth and young adults. Psychiatric Rehabilitation Journal 2012; 35(3): 167–170.
Burke-Miller J, Razzano L, Grey D, et al. Supported employment outcomes for transition age youth and young adults. Psychiatric Rehabilitation Journal 2012; 35(3): 171–180.
McKay C, Osterman R, Shaffer J, et al. Adapting services to engage young adults in ICCD clubhouses. Psychiatric Rehabilitation Journal 2012; 35(3): 181–188.
Bickman L, Heflinger CA. Seeking success by reducing implementation and evaluation failures. In: LB Bickman, D Rog (Eds). Children’s Mental Health Services; Research, Policy and Evaluation. Thousand Oaks, CA: Sage, 1995, pp. 171–205.
Brannan AM. Ensuring effective mental health treatment in real-world settings and the critical role of the family. Journal of Child and Family Studies 2003; 12: 1–10.
Behrens T, Foster-Fishman P. Developing operating principles for systems change. American Journal of Community Psychology 2007; 39: 411–414.
Parsons B. The state of methods and tools for social systems change. American Journal of Community Psychology 2007; 39: 405–409.
Kreger M, Brindis C, Manuel D, et al. Lessons learned in systems change initiatives: benchmarks and indicators. American Journal of Community Psychology 2007; 39: 301–320.
Ackoff R, Rovin S. Redesigning Society. Stanford, CA: Stanford Business Books, 2003.
Foster-Fishman P, Nowell B, Yang H. Putting the system back into systems change: A framework for understanding and changing organizational and community systems. American Journal of Community Psychology 2007; 39: 197–215.
Foster-Fishman P, Behrens T. Systems change reform: Rethinking our theories, methods, and efforts in human services reform and community-based change. American Journal of Community Psychology 2007; 39: 191–196.
Lindblom C. The science of “muddling through.” Public Administration Review 1959; 19: 79–88.
Emshoff J, Darnell A, Darnell D, et al. American Journal of Community Psychology 2007; 39: 255–267.
Foster-Fishman P, Droege E. Locating the system in system of care. Evaluation and Program Planning 2010; 33: 11–13.
Brannan AM, Brashears F, Gyamfi P, et al. Implementation and development of federally-funded systems of care over time. American Journal of Community Psychology 2012; 49: 467–482.
Hodges S, Ferreira K, Israel, N. “If we’re going to change things, it has to be systematic:” Systems change in children’s mental health. American Journal of Community Psychology 2012; 49: 526–537.
Morrissey J, Johnsen M, Calloway M. Evaluating performance and change in mental health systems serving children and youth: Interoroganizational network analysis. Journal of Mental Health Administration 1997; 24(1): 4–7.
Davis M, Koroloff N, Johnsen M. Social network analysis of child and adult interorganizational connections. Psychiatric Rehabilitation Journal 2012; 35(3): 265–272.
Walker J, Sanders B. The Community Supports for Wraparound Inventory: An Assessment of the Implementation Context for Wraparound. Journal of Child and Family Studies 2011; 20(6): 747–757.
Walker J, Koroloff N. Grounded theory and backward mapping: Exploring the implementation context for wraparound. Journal of Behavioral Health Services and Research 2007; 34(4): 443–458.
R Core Team. A language and environment for statistical computing. Foundation for Statistical Computing. Available online at http://www.R-project.org/. Updated 2013. Accessed February 18, 2014
Raudenbush SW, Bryk AS. Hierarchical Linear Models. Second Edition. Thousand Oaks, CA: Sage Publications, 2002.
Snijders TAB, Bosker RJ. Multilevel Analysis: An Introduction to Basic and Advanced Multilevel Modeling, 2nd Edition. London: SAGE, 2012.
Honaker J, King G, Blackwell M. Amelia II: A program for missing data. Journal of Statistical Software 2011; 45(7): 1–47. Available online at http://www.jstatsoft.org/v45/i07/. Accessed February 18, 2014.
Rosseel, Y. Lavaan: An R package for structural equation modeling. Journal of Statistical Software 2012; 48(2): 1–36. Available online at http://www.jstatsoft.org/v48/i02/. Accessed February 18, 2014.
Chen F, Curran PJ, Bollen KA, et al. An empirical evaluation of the use of fixed cutoff points in RMSEA test statistic in structural equation models. Sociological Methods & Research 2008; 36: 462–494.
Bentler PM, Bonett DG. Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin 1980; 88: 588–606.
Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling 1999; 6: 1–55.
Barr DJ, Levy R, Scheepers C, et al. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language 2013; 68: 255–278.
Acknowledgments
This work was supported by funding from the National Institute of Disability and Rehabilitation Research, United States Department of Education, and the Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, United States Department of Health and Human Services (NIDRR grant H133B090019). The content does not necessarily represent the views or policies of the funding agencies. The authors would like to thank the stakeholders who provided their insights and feedback for this work, as well as those who participated in the research study.
Conflict of Interest
The authors report no conflicts of interest.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Walker, J.S., Koroloff, N. & Mehess, S.J. Community and State Systems Change Associated with the Healthy Transitions Initiative. J Behav Health Serv Res 42, 254–271 (2015). https://doi.org/10.1007/s11414-014-9452-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11414-014-9452-5