Child and Adolescent Social Work Journal

, Volume 29, Issue 3, pp 167–184

Development and Evaluation of the Trauma-Informed System Change Instrument: Factorial Validity and Implications for Use

Authors

  • Margaret M. Richardson
    • Western Michigan University
    • The Evaluation CenterWestern Michigan University
  • James Henry
    • Western Michigan University
  • Connie Black-Pond
    • Western Michigan University
  • Yvonne Unrau
    • Western Michigan University
Article

DOI: 10.1007/s10560-012-0259-z

Cite this article as:
Richardson, M.M., Coryn, C.L.S., Henry, J. et al. Child Adolesc Soc Work J (2012) 29: 167. doi:10.1007/s10560-012-0259-z

Abstract

In this paper, we present the process of developing and evaluating an instrument designed to measure the extent to which a complex community system has changed as a result of a community initiative, and for purposes of this research, doing this within the content area of developing local trauma-informed child welfare systems in specific communities in Michigan. The instrument was designed for the Southwest Michigan Children’s Trauma Assessment Center’s (CTAC) Substance Abuse and Mental Health Services Administration (SAMHSA)-funded initiative to bring a trauma-informed perspective to professionals working with children in child welfare. The Trauma Informed System Change Instrument was developed with the input of experts in trauma-informed system change. Two parts of the instrument were analyzed separately using confirmatory factor analysis. A two factor model was fit for Community Characteristics and a three factor model for Individual Characteristics. Although adequate factorial validity was obtained for the instrument, specific items on the instrument that were problematic in fitting the model were identified, and suggestions for revising the instrument for improved functionality are offered, as are other potential uses of the instrument.

Keywords

Child welfareTraumaSystem changeInstrumentValidity

Introduction

By definition, children in the child welfare system have experienced maltreatment at the hands of their parents or other caregivers (Michigan Compiled Laws, 1975). Experiencing maltreatment within the family system has potential for long-lasting impact on a child’s neurological and emotional development (Cook et al. 2003). Children who have experienced abuse and neglect by the very people charged with their care are at risk for having chronic and ongoing trauma experiences (e.g., threat or perceived threat of serious harm or death, re-experiencing the events, hyperarousal, avoidance behaviors) (American Psychiatric Association 2000). For children, the experience of trauma impacts their development, emotions, and behaviors in a chronic, long-lasting manner (Van der Kolk 2005).

Experiencing maltreatment during early developmental growth creates neurological pathways that are adaptive to maltreating environments (Perry 2001). Removal of the child from these environments and placement with non-abusive caregivers may suggest that children will then adapt to the new, non-threatening environment. However, neurological changes have happened due to the need to survive the maltreatment, and moving the child to a safe environment does not logically lead to reduction in adaptive behaviors by the child, as structural adaptations in brain development have occurred (Teicher 2003). Without understanding this, conflict can arise in the foster or relative homes that provide placement for these children, as expectations are for children to simply adapt to what appears to be a nonthreatening environment, and for children to “appreciate” the new safe home they’ve been given.

These expectations are not in line with what trauma theory would tell us. Trauma theory suggests that children undergo physiological changes due to their traumatic experiences and these changes are enduring and long-lasting (Cook et al. 2005). Neurological differences due to trauma impact a child’s ability to regulate their emotional experiences, so that they are likely to have low tolerance for frustration and stress. They may become excessively labile (i.e., angry, crying, laughing) or excessively withdrawn and affectively flat. They may experience difficulty in modulating their emotional reactions and their behaviors (Schore 2003; Van der Kolk 2005). Attention and memory are likely going to be delayed (Henry et al. 2007; Perry 1999). When adult expectations are particularly high, or come in the way of a more traditional authoritarian perspective, this makes greater stress for the child when the child has difficulty modulating stress to begin with. Increased frustration on the part of the child may increase lack of control over behavior, which, in the traditional sense, increases parental response to control the child. For a traumatized child and their caregiver, this can become a vicious cycle that ends in frustration and resentment for both child and caregiver. It is this cycle that can exacerbate the potential for abusive behaviors in controlling the child (Harris et al. 2007). For children in foster care placements, this means placement changes as families cannot manage the traumatic symptoms, children fail in school as there are consequences of traumatization that inhibit school success, and children who have developed survival behaviors may act out in such ways that break the law or put self/others at risk (Ford et al. 2007). All of this is at an immeasurable human and financial cost to society.

A child welfare system that is not trauma informed likely misunderstands the child’s experience from the child’s view of the world. Instead, it involves treatment of the child from an adult perspective that does not consider the impact of trauma. Projecting an adult’s view of what might constitute traumatic experience onto the child may misinterpret the true experience of the child. This in turn sets up the relationship for misunderstanding, misinterpretation of intent, and consequences not suited to the etiology of the behavior (Heide and Solomon 2006).

Evidence-Based Trauma Informed Treatments and Interventions

In the last decade, there has been increased focus not only on treatment of childhood traumatic stress, but on the development of evidence based practices to treat traumatized children (Ford et al. 2007; Stambaugh et al. 2007). The National Child Traumatic Stress Network (NCTSN) has been instrumental in fostering understanding of the impact of traumatic stress on children:
  1. 1.

    Trauma Focused Cognitive Behavioral Therapy (TF-CBT, and forms of TF-CBT specific to grief and to sexual abuse) are evidence based treatments, as is Parent Child Psychotherapy, C-Bits, TARGET, and Real Life Heroes. All of these NCTSN-sponsored treatments have a different focus and a different target population, and demonstrate efficacy for treatment of their population within the treatment model. Many of these treatment models include a family component, or have family interaction integral in the treatment model; however, such mental health treatment, even when inclusive of the family system, is still delivered independently without mandatory involvement of the larger system (school, court, community). Even if a treatment is efficacious for a child’s symptoms over the short term, if the context in which the child functions does not have an understanding of the effects of traumatic stress on a child’s functioning, the positive changes may dissipate as the larger system will not support change. Systems theory informs on this point, as a system will always work to maintain equilibrium, and when one agent of the system changes, the other agents in the system will pressure the system to returning to the old and known methods of interaction (Williams and Imam 2007).

     
  2. 2.

    System interventions have been introduced by the NCTSN specifically for the child welfare system (Taylor and Siegfried 2005). Toolkits for training child welfare workers and resource parents (foster parents and relatives who serve as placement for children removed from their parents’ care) have been developed and used nationally. No multisite evaluation data exists for the effectiveness of these trainings or of the training material, although anecdotal information and individual site training evaluations suggests positive reception for the information. Although these efforts go a long way to address the need for system professionals to understand the impact of traumatic stress on children, in no way are these toolkits a comprehensive multiagency whole system intervention.

     

A Trauma Informed Child Welfare System

Awareness and acknowledgement that maltreated children have experienced significant trauma is increasing throughout the children’s services world, and if asked, there would be few children’s services professionals who would deny that most children entering foster care have experienced trauma. It is the understanding of the impact of traumatic experiences that has been lacking. This is harder for professionals and caregivers to accept, as it runs contrary to the medical model as explained above as well as traditional child rearing practices. Even when these models are ineffective in dealing with these children, the idea of achieving and maintaining equilibrium, as systems theory reminds us, is a powerful force. In practice, this means that change is hard and takes conscious considerable effort to institutionalize. It explains why, even when a practice and a paradigm of understanding these children is ineffective, and even harmful, people, and systems, continue the same actions as always. It is a long process to allow a paradigm shift to take hold.

The need for a trauma-informed child welfare system becomes apparent when viewing the child from the systems perspective, and as an agent interacting with many other, overlapping, agents. Then imagine the multiple systems surrounding multiple children, as well as each entity within the system (i.e., the court, a counselor, or a child welfare worker) surrounded by systems of interaction with multiple children, and the complexity of the system grows exponentially. So does the impact of positive change in just one of the entities, as well as the force of seeking equilibrium that happens when change enters a system. In other words, when a catalyst of change enters a system, a ripple effect can occur, so that this catalyst creates further change. As system theory predicts, other system forces (i.e., within the agency, from other agencies, from individuals in the system) will be in play to keep the change from happening in order to maintain equilibrium. If the forces for change are strong enough and come from enough entities within the system, especially entities with strength (power to change systems), the point of equilibrium can shift further in the direction of change to the new paradigm shift.

To make this process specific to children in child welfare, imagine the following vignette: a therapist who has been trained in trauma-informed practices is conducting TF-CBT with a 13 years old girl, and the child is making positive changes in understanding how her behaviors are the manifestation of traumatic experience. The girl, following the advice of her therapist, tries to calm herself when she is challenged or frustrated by other kids in the foster home by withdrawing to her room for a time. However, the foster parents believe that children should do as they are told, and insists that she join on family outings and every dinnertime, pointing out that she is given free time after dinner and chores are completed. Although these are reasonable demands for a non-traumatized child, the rigidity of the environment is not allowing the space for this child that she needs to learn to manage her own behavior and emotion. What she is learning in therapy does not seem to apply to the real world. The foster parents reject the suggestions by the counselor, pointing out that they cannot give special treatment to one child or they would all ask for special treatment. The caseworker supports the foster parents as she does not want the placement to break, and she joins with them in trying to impress to the child that she is safe in this home and needs to abide by the quite reasonable family rules.

In this vignette, there is an agent of change (therapist) who understands the etiology of the child’s behavior, but her efforts are squelched under the forces of the existing system. Taking the same example, imagine that the caseworker was also equally aware that the child needed specific interventions and day-to-day treatment that accounted for traumatic stress, but the foster parents were persistently resistant to this notion. Although the therapist and caseworker would have mutual support, if the treatment in the foster home was not conducive to healing from traumatic stress, the efforts for change for these two professionals may not be enough to meet the ultimate goal—reduction of traumatic stress symptoms (emotions, behaviors) as an indicator of the child’s healing from traumatic experiences. This is especially true if the court personnel are not attuned to the needs of a maltreated child, which is more often than not the case.

This vignette illustrates the need for multiple players in the child’s life to be on the same page in understanding the child’s needs. The case example given would be further complicated if court decisions that were not based on understanding the effects of traumatic stress were given, or if examples of non-trauma-informed school interactions were given. The example given is the plight of the majority of children in the child welfare system. It illustrates that, in spite of the best work being done to understand traumatic stress in children and to educate child welfare workers, that without an infusion that considers the county (jurisdictional site) as the unit of intervention, as opposed to individual “lone ranger” practitioners, system change will be less likely.

The National Child Traumatic Stress Network (NCTSN) has developed a conceptual framework entitled the Essential Elements (National Child Traumatic Stress Network (NCTSN) 2007), which detail essential components for any trauma-informed child welfare system. These essential elements cover relationship-building, creation of emotional safety for the child, skill-building and processing of trauma, addressing how trauma has a “ripple” effect in a child’s life, coordination between agencies, and the impact of working with trauma on the professional who manages these cases. The essential elements transcend roles in child welfare and are applicable to the system at large. They offer a valuable theoretical framework, but two issues relevant to the current project remain unaddressed by the essential elements, or elsewhere in the literature: (1) the elements have not been applied at a system level, with the system as the unit of change, and (2) they have not been operationalized satisfactorily at any level for purposes of valid and reliable measurement, let alone at the system level.

Establishing validity of a measurement for tracking change in a complex system relies on the ability (a) to define just what that system is, and (b) just how that system functions. Specifically, questions under the first point could be: what is the content area of the specific system, and to what extent does this content exist within the system? Under the second point, understanding how systems change can inform the understanding of a child welfare system can change to become more trauma-informed. One theory of change within complex systems suggests that change occurs in varying degrees in five major domains—context, components, infrastructure, connections, and scale (Coffman 2007). Three of these components are most relevant to change in child welfare systems:
  1. 1.

    Context. Local, state, and federal policy that shapes the focus and action of professionals throughout the local child welfare system. This includes the policy of each entity, including schools, foster care, courts, mental health, and policy shaping local collaborative bodies, for example

     
  2. 2.

    Components. This could refer to specific treatments or resources available locally that support a trauma-informed system, and these could look different depending on local context (e.g., evidence based practices, local child abuse prevention programs that address trauma, private agency work, school or law enforcement programs, fund raising efforts). Could also refer to day to day practice changes by individuals

     
  3. 3.

    Connections. Interagency commitment of time and resources to shared case planning for children, including shared financial responsibility, joint treatment planning, and policy planning at an administrative level

     

The challenge comes in determining to what extent change needs to occur in each of these domains, especially when considered within local contexts, which have unique problems and characteristics that call for unique program design. These are program theory of change issues, and it may be that the path to change may be different for different sites, but the ultimate destination looks very similar for every site. For purposes of this paper, this means that process may vary according to context, but outcome measurement, the purpose of the instrument being developed, will remain constant regardless of context. In other words, the essential elements of change in a complex system may be operationalized in such a way that they hold meaning across contexts and are true constructs of paradigm shift in a complex system, in this case, a trauma-informed child welfare system. Given this, the items in the Agency Practice section of the Trauma Informed System Change Instrument were developed from the hypothesized system domains of Policy, Components, and Connections.

Statement of the Problem

The realization of a need for an instrument to measure the impact of system change initiatives, in this case, moving toward a trauma informed paradigm, came from grassroots need at the community level, and as well from the growing need in the NCTSN community of grantee sites. More agencies around the country are engaging in the work of changing systems, and evaluation of their efforts has not kept pace with the need. The realization for need of intervention at a larger systemic level has come after pallid results in focusing just on the service provider level. The need to engage a community to support a paradigm shift became evident. Although there has been understanding of the need at a system level, training and intervention in work with traumatized children has still been focused at the level of the individual and the service provider. The initiative conducted through the Southwest Michigan Children’s Trauma Assessment Center (CTAC) is designed to engage the leaders of key agencies within the community (courts, Department of Human Services, schools, and mental health) and to support the efforts of grass-roots champions for trauma informed change.

Measurement of trauma informed change in child welfare has been conducted regarding service provider change or individual change, but not of perception of change in the function of the agency within the child welfare community. Any evaluation efforts of such a system change initiative are in the early stages at best, and the need for an instrument to help produce evidence to support the process has become clear. It is from this need that an instrument to fill the gap was developed.

Development and Pilot-Testing of the Trauma Informed System Change Instrument

The design of this instrument stemmed from an applied social work initiative, so that instrument development occurred not only within the process of designing the initiative evaluation, but within the actual designing of the initiative itself. The first iteration of the instrument was a product of this developmental process. To serve the project, the first phase of this study—the developmental period for instrument design—was abbreviated due to the need to put the instrument in the field for use.

It was after the pilot of the instrument that the role of individual change was considered and added as part two of the instrument. Individual factors in adopting a change in professional perception and practice were hypothesized to mediate the extent to which an individual perceives change in the larger system. Aarons (2005) researched the adoption of evidence-based practices and proposed four domains important for the adoption of new evidence-based practice: requirement to take on the practice, appeal of the practice, openness to new ways of doing things in general, and divergence between current practice and perceived differences with the new practice. The openness scale from Aarons’ Evidence Based Practices Attitude Scale was used as a starting point to structure questions regarding the openness to take on trauma informed practice.1 In addition, items were developed to gauge the extent to which individuals perceived themselves as integrating trauma understanding into their practice. This resulted in two subscales—Integration and Openness—for the Individual Practice section of the Trauma Informed System Change Instrument.

Research Questions Investigated

The following broad research questions were investigated in relation to the Trauma Informed System Change Instrument:
  1. 1.

    To what extent is there evidence to support factorial validity of the Trauma-Informed System Change Instrument?

     
  2. 2.

    To what extent is the hypothesis of six latent constructs (three related to individual characteristics and three related to system characteristics) in the Trauma-Informed System Change Instrument supported?

     
  3. 3.

    To what extent are items in the Trauma-Informed System Change Instrument internally consistent?

     
  4. 4.

    To what extent is the Trauma-Informed System Change Instrument useful for measuring change over time?

     

Method

Sample

The Trauma-Informed System Change Instrument was administered to 342 persons in four Michigan communities between 2008 and 2009 as part of the CTAC training initiative. A request for completion of a 3-month follow-up was sent to all of the baseline respondents who provided their contact information (N = 130) and a reminder was sent 1 week later. Thirty-three persons completed the follow-up for a response rate of 25.3%. Participants consisted of court personnel, Michigan Department of Human Services (DHS) personnel, school personnel, mental health professionals, and medical personnel.

Instrumentation

The Trauma-Informed System Change Instrument consists of 19 Likert-type items scaled from 1 to 5 and anchored at 1 = not at all true, 3 = somewhat true, and 5 = completely true. The original 26 item instrument consisted of six hypothesized latent factors: (1) Policy; (2) Components (renamed Agency Practice); (3) Connections; (4) Integration; (5) Openness; and, (6) Tradition. After analysis, the remaining 19 items were fit into four latent factors: (1) Policy; (2) Agency Practice; (3) Integration; and (4) Openness.

Procedures

Children’s Trauma Assessment Center training participants were provided an opportunity to complete the instrument after an orientation session in which a trauma-informed child welfare system was discussed and conceptualized. Participants were informed about the CTAC initiative through local community channels (i.e., agency meetings, e-mails from a local community “champion,” and newspaper articles or newsletters). Attendance was not limited to particular agencies or types of professions, and included a variety of professionals and resource parents who interacted in some way with the child welfare system.

Data Processing and Analysis

Of the 342 original cases, 28 (8%) were eliminated due to missing responses (missingness ranged from .9% on several items to 7.3% on one item, for an average of 2.5% across all items) that were determined not to be missing at random (MAR), leaving a total of 314 (92%) cases for analysis. Given the nature (not MAR) and proportion (i.e., 8%) of missingness, no imputation methods were used to replace missing values. In general, pairwise deletion is not recommended, and listwise deletion is problematic unless the missing data have been demonstrated to be missing MAR (Schreiber et al. 2006).

Confirmatory factor analysis (CFA) was used to test the hypothesized factorial structure of the Trauma Informed System Change Instrument separately for Individual Characteristics and Community Characteristics, given that the two are considered independent aspects of trauma-informed system change. The parameters and fit of the CFA models were estimated from the raw data and sample variance–covariance matrices using maximum likelihood (ML) and weighted least squares mean and variance adjusted estimators (WLSMV; see Beauducel and Herzberg 2006, for a comparison of the properties of ML versus WLSMV) methods in Mplus 5.1 (Muthén and Muthén 2007). ML was used to test competing, nested models and WLSMV was used to compare fit of the ML final models given the ordinal nature of the data (WLSMV does not allow testing of alternative models due to the way in which degrees of freedom are calculated). In part, the use of both ML and WLSMV methods of estimation were applied as an informal sensitivity analysis to examine the robustness of the models under different distributional assumptions.

Model fit was evaluated using X2 difference tests (∆X2) for nested models as well as X2 goodness-of-fit, the X2/degrees of freedom ratio (X2/df), the goodness of fit index (GFI), the comparative fit index (CFI), the root-mean-square error of approximation (RMSEA) and its 90% confidence interval (90% CI), the Tucker-Lewis Index (TLI), Hoelter’s Critical N (CN), and expected cross-validation index (ECVI) for all models. Multiple indices were selected because they provide different information for evaluating model fit (i.e., absolute fit, fit adjusting for model parsimony, fit relative to a null model). Used together, these indices provide a conservative and reliable estimate of model fit (Brown 2006).

Results

Research Questions #1 and #2: Factor Structure and Factorial Validity

Individual Characteristics

Four models of Individual Characteristics were tested. Model 1 (the hypothesized, baseline model) was a correlated three-factor model with 11 manifest variables, Model 2 was a correlated three-factor model with 9 manifest variables, and Model 3 was a correlated three-factor model with 8 manifest variables. Shown in Table 1 are ∆X2 tests for each nested model. The ∆X2 test of Model 2 and Model 3 did not indicate a statistically significant improvement in fit (∆X2 = 11.8, dfdiff = 7, p = .107). Given that one of the latent factors in Model 3 had only two manifest variables, which can be problematic (see Kline 2010), Model 2 was considered as the final model based on the analysis. However, the third latent factor, Tradition, was problematic in the analysis, based on low internal consistency, low factor loadings, and low programmatic utility of the questions, and was removed. Model 4 is a two-factor model with 6 manifest variables.
Table 1

Comparison of competing models for individual characteristics

 

X2

df

X2/df

X2

GFI

CFI

TLI

RMSEA

CN (p = .05, p = .01)

Model 1

210.4

41

5.132

NA

.898

.822

.762

.115

85, 97

Model 2

50.6

25

2.024

159.8*

.967

.968

.953

.057

233, 275

Model 3

38.8

18

2.156

11.8

.971

.968

.953

.061

233, 281

Model 4

8.3

6

1.387

42.3*

NA

.997

.991

.035

474, 633

p < .05

For the final Individual Characteristics model (i.e., Model 4) using ML estimation, 6 parameters were estimated. The model was overidentified with dfM = 6, and a converged, admissible solution was obtained. Overall, the model adequately reproduced the data (X2 = 8.3, df = 6, p = .001; X2/df = 1.387; CFI = .997; TLI = .991; RMSEA = .035, 90% CI = .000–.087; CN = 474 for p = .05 and 633 for p = .01; ECVI = .161 for the tested model and .173 for the saturated model). The model fit was evaluated as adequate, and the null hypothesis of no difference between the model-implied population covariances and the observed sample covariances could not be rejected based on the X2 goodness-of-fit test. The standardized residuals and modification indices indicated no localized points of ill-fit in the solution and all freely estimated unstandardized parameters were statically significant at p ≤ .05. The unstandardized estimates and standardized parameter estimates for the final two-factor Individual Characteristics model are shown in Fig. 1. As shown in the figure, the interfactor correlation between the latent constructs Integration (ε1) and Openness (ε2) was ϕ = .47.
https://static-content.springer.com/image/art%3A10.1007%2Fs10560-012-0259-z/MediaObjects/10560_2012_259_Fig1_HTML.gif
Fig. 1

Final model for individual characteristics. Note: Unstandardized and standardized parameters with standardized parameters shown in parentheses

The final correlated two-factor Individual Characteristics model was retested using the WLSMV method of estimation and parameter estimates and goodness-of-fit indices (X2 = 28.9, df = 6, p > .000; CFI = .983; TLI = .983; RMSEA = .11) were reasonably comparable to the ML model; however, the RMSEA nearly doubled from the ML RMSEA of .057. This model was not cross-validated on another sample as this portion of the instrument had not been piloted elsewhere.

Community Characteristics

Four models of Community Characteristics were tested. Model 1 (the hypothesized, baseline model) was a correlated three-factor model with 17 manifest variables, Model 2 was a three-factor model with one second-order factor, 17 manifest variables, and one correlated error, Model 3 was a two-factor model with one second order factor, 13 manifest variables, and two correlated errors, and Model 4 was a correlated two-factor model with 13 manifest variables and two correlated errors. Models 2 and 3, however, yielded nonadmissible solutions. All fit indices and the ∆X2 tests for Models 3 and 4 are identical. However, the models differ in that the correlation between the two latent factors in Model 4 was replaced with a second order factor accounting for the variance in Model 3. As shown in Table 2, Model 4, a nested model of both Model 1 and Model 2, represented a statistically significant improvement in overall model fit (∆X2 = 170.6, dfdiff = 24, p > .000).
Table 2

Comparison of competing models for community characteristics

 

X2

df

X2/df

X2

GFI

CFI

TLI

RMSEA

CN (p = .05, p = .01)

Model 1

445.2

87

5.117

NA

.835

.860

.832

.115

78, 85

Model 2

295.7

86

3.438

149.5*

.885

.918

.900

.088

116, 127

Model 3

125.1

62

2.018

170.6*

.943

.971

.964

.057

204, 228

Model 4

125.1

62

2.018

170.6*

.943

.971

.964

.057

204, 228

p < .05

For the final Community Characteristics model (i.e., Model 4) using the ML method, 29 parameters were estimated. The model was overidentified with dfM = 62, and a converged, admissible solution was obtained. Overall, the model adequately reproduced the data (X2 = 125.1, df = 62, p > .000; X2/df = 2.018; GFI = .943; CFI = .971; TLI = .964; RMSEA = .057, 90% CI = .042–.071; CN = 204 for p = .05 and 228 for p = .01; ECVI = .585 for the tested model and .581 for the saturated model). As with the Individual Characteristics model, although model fit was evaluated as adequate, the null hypothesis of no difference between the model-implied population covariances and the observed sample covariances could not be rejected based on the X2 goodness-of-fit test. The standardized residuals and modification indices indicated no localized points of ill-fit in the solution and all freely estimated unstandardized parameters were statically significant at p ≤ .05. The unstandardized estimates and standardized parameter estimates for the final two-factor Community Characteristics model are shown in Fig. 2. As shown in the figure, the interfactor correlation between the two latent constructs Policy (ε1) and Agency Practice (ε2) was ϕ = .86.
https://static-content.springer.com/image/art%3A10.1007%2Fs10560-012-0259-z/MediaObjects/10560_2012_259_Fig2_HTML.gif
Fig. 2

Final model for community characteristics. Note: Unstandardized and standardized parameters with standardized parameters shown in parentheses

The final correlated two-factor Community Characteristics model was retested using the WLSMV method of estimation and parameter estimates and goodness-of-fit indices (X2 = 100.4, df = 36, p > .000; CFI = .971; TLI = .991; RMSEA = .075) were reasonably comparable to the ML model; however, the RMSEA increased slightly from the ML RMSEA of .057. This model was cross-validated on a second sample (N = 131), producing similar, though not equivalent, results (see Richardson 2010, for further details regarding the WLSMV models and cross-validation).

Research Question #3: Internal Consistency

In addition to Cronbach’s α, ordinal α and ω reliability coefficients were estimated for each of the five latent constructs by using a polychoric correlation matrix to account for the ordinal nature of the variables (Gugiu et al. 2009, 2010; Zumbo et al. 2007). Estimates for Cronbach’s α, ordinal α, and ordinal ω for the five latent factors are shown in Table 3. Overall, the constructs exhibit reasonable internal consistency, with the exception of the Tradition factor (Cronbach’s α = .527, ordinal α = .654, and ordinal ω = .679), which had subsequently been removed from the instrument.
Table 3

Comparison of reliability coefficients

Latent factor

Cronbach’s α

Ordinal α

Ordinal ω

Policy

.869

.874

.887

Agency practice

.875

.883

.890

Integration

.847

.851

.863

Openness

.735

.787

.801

Tradition

.527

.654

.679

Research Question #4: Pretest–Posttest Effect

For the subsample of participants who were administered the Trauma-Informed System Change Instrument prior to (i.e., pretest) and following (i.e., posttest) CTAC training, effect sizes in the form of pretest–posttest standardized mean differences were estimated using both Cohen’s d and Hedges’ g (given that d tends to overestimate the value of the population parameter δ in small samples). The 3-month pretest–posttest effect following CTAC training on Policy was Cohen’s d = .365 and Hedges’ g = .359 and on Agency Practice was Cohen’s d = .363 and Hedges’ g = .357. All effects were statistically significant (p < .05).

Discussion

This study originated from the need to measure the extent to which child welfare systems were becoming trauma informed during and after participation in a community-wide training initiative. Necessity fueled the construction of an instrument to fill this purpose, as there are no existing methodologies or instruments to document either to what extent child welfare systems are functioning in a trauma informed manner, or to what extent are these systems changing after efforts are made to transform functioning of the systems. Professionals from CTAC had developed an initiative to help communities understand what it means to be a trauma informed child welfare system, identify the need for trauma informed change in their community, address the identified need, and take necessary steps to integrate and sustain these changes over the long term. As this type of initiative is in its infancy nationally, the CTAC professionals are among the vanguard for creation of trauma informed change, and are poised in a favorable position to provide consultation and feedback on content when developing evaluation methods to chart the change and, specific to this work, insight into the development of items for an instrument to document current community status and change over time.

The instrument was developed with a systems perspective, and through review of existing literature, three areas of systems were defined for measuring change: Policy, Agency Practice, and Connections (between individuals and between agencies). Content experts were asked to support item development for system change specific to the trauma informed initiative. After piloting, the instrument was revised. Additional measures of individual change were developed based on an existing instrument. These items measured the attitude of adopting new practice, as it was hypothesized that controlling for individual attitudes regarding change in practice would be vital in determining the extent to which communities were changing.

The revised instrument was used for administration to cohort two participants. It was through the abundance of responses from this series of administrations of the instrument that the determination of factorial validity of the instrument became a distinct possibility, as too few respondents would have precluded this level of analysis. A spirit of frugality in preserving usable data drove the design of the current study, so that cohort one data could be used to answer research questions concerning impact of the initiative, and specific characteristics of the instrument in determining impact (sensitivity of the instrument, the fit of the instrument in the overall evaluation), and cohort two data could be used for determination of psychometric properties of the instrument (as well as impact of the initiative at a later date).

Confirmatory factor analysis was used to fit a factor structure to the existing data. For Community Characteristics, a two factor model was fit for 13 items. Goodness of fit indices were excellent, as were parsimony indices and the index of adequacy of sample size to fit a valid model. Evidence of convergent validity was good in that the standardized estimates were relatively large for each set of indicators. However, the interfactor correlation is quite large so that support for divergent validity is not strong. Such a high interfactor correlation suggests a second order factor structure (Bollen and Long 1993), but this was tested and invalid results were obtained. Internal consistency was found to be adequate. For Individual Characteristics, a three factor model was fit for 9 items initially. However, all three items in the third factor cross-loaded onto other factors, internal consistency for this factor was poor, and it had a very low factor loading. Removal of the third factor slightly improved goodness of fit, so that the final model fit 6 items to two factors. Goodness of fit indices and parsimony indices were adequate, and although nested models were tested, alternative hypotheses for factor structures were not necessary.

Using the factor structures, paired t-tests were conducted on the pretest–posttest data received from cohort one. Although the number of paired responses was small, there were enough to show statistically significant change and detection of a moderate effect size. Results from use of the instrument were consistent with evaluation results obtained through observations and interviews.

Limitations of the Present Research

In spite of obtaining valid factor structures for both sections of the instrument, and for documenting statistically significant change in the impact study, there are limitations to the study.

Contextual factors greatly impact the analysis for evaluation of this instrument. Defining the “child welfare system” is different for different communities, and catchment areas for services are different for different communities, and the impact of this difference potentially muddies the salience of positive results. The catchment area for agencies differs from community to community, and this is not easily defined so to be able to account for introduction of bias and mediating effects in each area. For example, the community mental health agencies, Department of Human Service agencies, and Intermediate School Districts do not consistently correspond in the counties they represent. Therefore, defining the child welfare system for each participant is very much context-dependent, and each community’s organizational responsibilities and barriers are unique and complex. To compare one individual’s responses in how they communicate with other agencies to another individual in an adjacent community could be comparing apples and oranges, as each individual may be referring to different agency players. To further complicate matters, HSIRB consents for participation made it clear that identifiers do not need to be present on completed evaluation forms, so that from where a participant hales often was unknown. Although participants are united by one broad purpose—child welfare, the sample numbers were not large enough to power more sophisticated statistical analyses that could have taken into account differences between communities.

Aspects of conducting research in context bring limitations. The heterogeneity of the sample introduces a significant source of potential variability into responses. Because this is a community sample and the respondents did not need to identify themselves, there may be very different professional perspectives being represented in the responses (e.g., a police officer completing the instrument is more ancillary to the child welfare system and will likely see things very differently than a therapist, even if they work with the same systems and with the same children). The problem comes in not being able to identify and pinpoint these potential sources of bias and variability consistently.

Also, conducting the research within an applied social work setting precluded the availability of a comparison group to account for threats to internal validity. As the training team delivering the initiative has done trainings all across the state of Michigan, most child welfare communities have had some exposure to the trauma trainings, although not to the systematic child welfare initiative.

There are limitations due to sample size and nonnormality of the data. The sample size was too small to allow for a test–retest reliability study. The nonnormal skewness is less important for CFA because it is not a mean-based analysis, but kurtotic data does have an impact. Several items on the instrument were kurtotic, but two of the items with greatest kurtosis were removed. The extent to which other kurtotic items impacted the analysis is not fully known. Implicit in kurtosis is the idea of ceiling and floor effects perhaps a more sensitive scale, such as a Likert scale up to 7 or 9 would have helped to alleviate the ceiling/floor effects.

Finally, as this is essentially ground-breaking work in trauma informed system change for child welfare, which brings with it a sense of excitement and possibility, it also means that there is no other work to compare to in order to build a case for criterion validity. Work has not been done in this area to the extent to begin considering predictive validity.

Conclusion

The current study is an important step in establishing measurement of change in child welfare systems as they begin to incorporate trauma informed practice and understanding of children. Once a construct is measured, it becomes “real” in the sense that it has been operationalized and is visible. The impact of trauma is not currently mentioned in the Federal Standards for child welfare, and treatment is referred to in the more general term of “appropriate.” As the initiative is impacting professionals in the participating communities in Michigan, the next step in the program theory is that the children will begin to show impact. If trauma informed practices are the way to mitigate symptomology for traumatized children, and if professionals are willing and able to conduct their work with children in trauma informed ways (including as an agent within a system), then symptomology for traumatized children should reduce. As measurement of symptomology is one proxy for child well-being, it follows that child well-being should improve with reduction of symptoms. The development of this instrument plays a key role in this. Having an instrument that has been evaluated for validity in measuring change in child welfare is key to supporting the program theory. In the attempt to measure the construct of “trauma informed,” the current instrument falls short as was described in the previous chapter; however, it is a step in the right direction, and builds a foundation for future work to operationalize “trauma informed” within a system framework.

The instrument has utility in measurement of trauma informed child welfare systems in three ways. First, it provides a snapshot of the extent to which the current community child welfare system is trauma informed from a cross-agency perspective. The wording of the instrument is purposefully universal so that it has meaning to individuals working in or with the child welfare system regardless of their role or agency affiliation. Second, it has value as a tool for targeting areas of need in developing an individualized training and consultation intervention to create a more trauma informed system. Finally, it can be used to show change over time regarding the extent to which the child welfare system is becoming trauma informed.

Because this is seminal work in trauma informed system change, there is really a universe of potential directions, levels of specificity, and ranges of focus that future research can take on. The most obvious area of future research is in the honing of the instrument for national use in trauma informed systems. Gaining the opinions and feedback from national experts in creating an instrument that is relevant to all types of communities, different venues (e.g., school focus versus court focus versus mental health focus), and for different purposes is vital. The use of the instrument by different experts in trauma informed practice and in different venues of trauma informed work will aid the generalizability of the instrument, or will inform the discussion as to whether another version of the instrument is needed. All of this input will inform content validity of the instrument in trauma informed venues, which is a necessary step in establishing validity. Reworking the instrument to attain greater predictive validity is vital to understanding the construct of being trauma informed. Determining whether there is a second order construct or more first order constructs will help to shape systems interventions, so that they more accurately target areas of change in systems. In other words, the development of better measurement is a reciprocal process with the development of better interventions to improve trauma informed system practice.

Footnotes
1

The Divergence Scale had initially been adapted and piloted as a “Tradition” subscale, but due to problems with utility and psychometrics, this adapted subscale was removed from the instrument.

 

Copyright information

© Springer Science+Business Media, LLC 2012