Introduction

Traumatic stress and trauma are pervasive public health concerns that impact the daily lives of many Americans (Bassuk et al. 2017; Magruder et al. 2017). More than 60% of American adults have experienced at least one adverse childhood experience (CDC 2019). Similarly, 60% of children experience at least one trauma each year (Finkelhor et al. 2005). Those rates increase when examining the global population, where 70% of people reported at least one traumatic event in their lifetime (Kessler and Ustün 2008). Rates of trauma exposure further increase when examining marginalized populations (Adams 2010; Alim et al. 2006; Bassuk et al. 1996; Goodman et al. 1997; Hatch and Dohrenwend 2007; Hayes et al. 2013).

Exposure to traumatic events can have long-lasting consequences, both physical and mental (Drury et al. 2012; Copeland et al. 2007; Felitti et al. 1998). However, evidence has demonstrated that people can overcome traumatic experiences with appropriate interventions and support (Covington 2008; Dozier et al. 1994; SAMHSA 2014). As a result, many researchers, clinicians, and legislators are looking for ways to not only prevent trauma but to interact with and support individuals who are traumatized. As the majority of both the US and global populations have experienced a traumatic event in their lifetimes, it is becoming increasingly necessary that individuals and organizations examine how their policies and procedures can impact individuals who have experienced trauma. Trauma-informed systems change has multiple layers and demands the support of leadership in any given system, adjusting policies and practices to focus on safety and reduce re-traumatization, and providing inter- and multidisciplinary training and coaching of members of a system (Oehlberg 2008).

Within the mental health field, the acknowledgment of trauma is evolving. As the public and mental health professionals are learning more about the importance of addressing trauma, trauma-informed care is becoming increasingly prevalent (SAMHSA 2014). However, there is an opportunity for mental health systems to more consistently integrate an understanding of trauma and its impact on the biopsychosocial framework as a means of improving treatment and outcomes, as this is not a general practice in mental health systems currently (Sweeney et al. 2018). For example, Hepworth and McGowan (2013) demonstrated that while many mental health professionals recognized the significance of trauma and the need to inquire about it, there is inconsistency in trauma inquiry during mental health assessments in acute settings, and even less consistency in routine inquiry about childhood sexual trauma in acute mental health settings. One study of general practitioners’ routine assessment of trauma found that they may feel reluctant to assess trauma for a variety of reasons, such as a lack of sufficient training in trauma-informed care and how to employ trauma-informed practices or a fear of causing harm when discussing a patient’s more complex issues (Tomaz and Castro-Vale 2020).

Importantly, research has demonstrated that trauma-informed interventions can improve attitudes and knowledge about trauma-informed practices and approaches and show promise for encouraging trauma-informed change on an individual and group level in both clinical and non-clinical (e.g., child welfare, school, business, government) settings (Damian et al. 2019; Kenny et al. 2017; Kramer et al. 2013; Lang et al. 2016; Niimura et al. 2019; Palfrey et al. 2019; William and Smith 2017; Haime 2020; SAMHSA 2020; Schreiber et al. 2006). Yet, as highlighted by Niimura et al. (2019), few studies have examined the effectiveness of trauma-informed care training programs using standardized measures with follow-up assessments. Even fewer have used measures validated for use across multiple systems (Champine et al. 2019).

Even outside of mental health, there is a desire to become more trauma-informed in fields ranging from dentistry to children’s sports leagues (Donisch et al. 2016; D’Andrea et al. 2013; Purtle 2020; Raja et al. 2015). However, as in health care fields, there is an absence of standardization in measuring the effectiveness of trauma-informed initiatives. A recent meta-analysis that included 33 studies on the implementation of trauma-informed practices and approaches in education pointed out that multiple disciplines have employed different methods in examining trauma-informed practices in schools, but examination of the impact of trauma-informed practices and approaches on educators is limited in current literature (Thomas et al. 2019). This analysis also showed limits to the effectiveness of school-based trauma-informed interventions because of the absence of evaluation and standardization of trauma-informed systems change.

As a further example, a 2019 systematic review of organization-wide implementation of trauma-informed initiatives in the child welfare system demonstrated that some form of trauma-informed training (each study had different dosages and content of training) was effective in positively impacting staff knowledge, skills and / or confidence. However, their analysis also showed weaknesses in study designs, and a lack of standardization or consistency in evaluation of service user, treatment, and training outcomes (Bunting et al. 2019).

There is also an effort to standardize trauma-informed training efforts for police officers and criminal justice professionals. Much of this work has been led by the Substance Abuse and Mental Health Services Administration (SAMHSA) GAINS Center, which created a trauma-informed response tool for criminal justice professionals with the stated goals of: (1) increasing understanding and improving awareness of trauma, (2) equipping members of the criminal justice system with trauma-informed tools and practices, and (3) providing a framework for trauma-informed policy change. This is an emerging training, and validation of the tool and its impact is yet to be determined. Equally, it is unclear how case studies that highlight cultural and gender issues (which is critical in any trauma-informed training model) is employed in the framework (SAMHSA, Trauma Training for Criminal Justice Professionals 2014). It is encouraging to see the criminal justice system focus on trauma-informed practices and approaches and move toward making trauma-informed change. However, consistent evaluation with validated measures is lacking, and it will be most important to be able to evaluate and measure the impact of trauma-informed systems change. A key part of measuring trauma-informed systems change is including cultural and historical components in a scale, such as the impact of racism and discrimination.

Meeting the need for a standardized and culturally responsive measure of trauma-informed systems change

Because trauma is ubiquitous and experienced widely in society across various systems (education, family, community, workplace, business, government, criminal justice, healthcare), it is imperative that every system promote a trauma-informed environment. It is clear from these examples that there is a desire to be trauma-informed, but there is no consistent framework on how to implement and/or measure trauma-informed systems change across fields. Moreover, core to trauma-informed principles are historical, cultural, and gender considerations. Many systems, including mental healthcare, have engaged with trauma-informed training, practices and approaches but were inadequate in considering historical and cultural factors.

Here we describe the Survey for Trauma-informed Systems Change (STISC), which was established to standardize the assessment of trauma-informed care, practices, and approaches in a multidisciplinary, cross-systems fashion (see Fig. 1). There is a lack of existing scales quantifying trauma-informed care that measure culturally responsive systems change. Some scales were designed for one specific field, such as education (e.g., ARTIC) (Baker et al. 2016) or health and human services (e.g., TIC Grade; TICOMETER, “Knowledge, Attitude, and Practice Related to Trauma-informed Practice” tool) (Bassuk et al. 2017; King et al. 2019; Sinko et al. 2020a, b), but these may not be generalizable. Because trauma impacts every system and field, there would be great utility in a scale that is generalizable across contexts (i.e., a comprehensive assessment of a trauma-informed, culturally responsive system that can be used in any system); however, a scale like this does not currently exist. To meet this need, we developed a scale (STISC) that could be administered to any professional, in any field, for assessment of trauma-informed and cultural responsivity of individuals and organizations. Availability of the measurement tool is critical to the development and evaluation of trainings and interventions aimed at improving trauma-informed practices across a broad range of systems and settings. This paper outlines the process of validating the STISC, a comprehensive assessment tool that measures trauma-informed, culturally responsive knowledge, attitudes, and practices across fields and systems.

Fig. 1
figure 1figure 1figure 1figure 1figure 1figure 1

Pre and post Survey for Trauma-informed Systems Change (STISC)

Full size image

Methods

Institutional Review Board (IRB) approval was granted by Massachusetts General Brigham, IRB2021P002889for evaluating the impact of trauma-informed training on a multidisciplinary, cross-sectional group in a virtual setting. This study had three main stages: (a) development of the survey used for pre- and post-training assessment; (b) administration of the survey to participants 24 hours prior to two-day, 12-hour interactive trauma-informed training workshops and within 48 hours of completion of the workshops; (c) preliminary psychometric validation of the survey as a tool to evaluate and measure trauma-informed, culturally responsive knowledge, attitudes, and practices.

Development of the survey for trauma-informed systems change

We used the methodology for scale development and validation of Boateng et al. (2018). Following this methodology, we began generating items for the survey. To begin drafting an instrument, the study team initially focused on knowledge and attitude-based questions related to trauma and cultural responsivity. In our review of the literature, we found that there were scales that already measured attitudes and perceptions related to trauma and trauma-informed care, but not necessarily behaviors and/or cultural responsivity (e.g., ARTIC, TICS-10) (Hales et al. 2019). As a result, we focused on developing a scale that measured practices and behaviors and cultural sensitivity in addition to knowledge and attitudes. We further found that current scales measuring trauma-informed change focus on specific systems, such as healthcare, social services (TIC Grade, TICOMETER) or child welfare (Trauma-informed Systems Change Instrument) (Richardson et al. 2012). Our goal was to design a measure that was applicable to all industries and professions.

The instrument was divided into four sections covering the four domains to be assessed by the scale. These sections are knowledge and attitudes; trauma-informed, culturally responsive practices in the workplace; assessing interactions with clients; and assessing safety and acceptance. Items corresponding to each section were chosen using SAMHSA’s key principles of a trauma-informed approach (safety; trustworthiness and transparency; peer support; collaboration and mutuality; empowerment, voice, and choice; cultural, historical, and gender issues), as well as their 10 implementation domains (governance and leadership; policy; physical environment; engagement and involvement; cross sector collaboration; screening, assessment, treatment services; training and workforce development; progress monitoring and quality assurance; financing; evaluation) (SAMHSA 2014). As suggested by Boateng et al. (2018) once questions for each section were drafted and reviewed as a study team, we reviewed our proposed scale with experts in the fields of trauma, psychology, and psychiatry. After discussing each item with leaders in the field of trauma research, they helped pare the scale down to 59 items that were appropriate, interpretable, and accurate. After the content of the scale was verified, we then administered the draft to multiple additional experts to receive feedback, administered in two rounds of feedback. From a qualitative perspective, we asked the target groups to verbalize their thought process when providing their answers to ensure that the questions were gathering the information we wished to capture. Before administering the scale to actual participants in trauma-informed trainings, we performed another round of mock-administration to determine how long the survey would take to complete. This was done by sending the link to the survey to [10] trauma researchers and having them complete the survey in its entirety.

Participants and procedure

Because the scale was created to generalize across industries and professions, we wished to include participants from a wide variety of fields for validation. Our participants were recruited from monthly trainings conducted by the Institute for Trauma-Informed Systems Change within McLean Hospital. These seven trainings took place between July 2021 and February 2022. All participants were sent an electronic pre-survey before the training. If the pre-survey was completed, the participant was sent a post-survey at the conclusion of training. In addition to the 59 items selected for the scale, the pre-training survey included questions on demographics (race/ethnicity, gender, and education level) and both the pre-survey and post-survey included a question on elements that define the respondent’s culture.

For survey administration with the target groups, participants were sent an individualized link to a REDCap survey before their scheduled workshop. The pre-survey expired on the day of their training before participants learned any content from the training. The 262 participants (including lawyers, doctors, government agencies, community-based organizations, fortune 500 CEO’s, healthcare, government and business leaders from the UK and Africa) received 12 hours of core trauma-informed teaching and training in two days on the fundamentals of trauma, fear, human development, trauma-informed systems change, the psychological impact of racism, and cultural responsivity. They were given tools and an active challenge to meaningfully apply the learning in making trauma responsive changes in their respective work systems. After the two-day, 12-hour training, participants that had completed a pre-survey received an individualized electronic link to the post-survey. All survey responses were anonymous, but the pre- and post-surveys were linked to each other via unique, nonidentifiable subject keys.

Statistical approach

Prior to analysis, items were assigned to one of seven subscales by the first author (AM) according to measurement domain and assessment of the domain at the individual (I), system (S), or individual-within-system (I-S) level: self-assessed knowledge and attitudes (I), system-wide knowledge and attitudes (S), training and employee support (S), interactions with clients (I-S), personal safety and acceptance at work (I), promotion of safety, acceptance and inclusion by system (S), and acceptance of cultural differences (I-S). Likert ratings for each item were assigned numeric values from 1 to 5 and summed to obtain subscale scores. To assess item performance, Spearman item-total correlations were calculated for each item and for each subscale. Items correlating substantially higher on an alternate subscale than their assigned subscale were candidates for reassignment. Items with no more than a weak correlation (< 0.30) with any subscale were candidates for removal or regrouping with other items. Subscales sharing a substantial number of items with moderate (> 0.40) or stronger correlation were considered for merging into a larger, more general subscale. All modifications of item assignments to subscales were undertaken with consideration of the theory underlying the scale’s development and the face validity and comprehensiveness of the subscales. Internal consistency reliability of the subscales based on original item assignment and final modified item assignments was quantified using hierarchical omega (Kelley and Pornprasertmanit 2016).

To evaluate the factor structure of the scale, the model corresponding to the final modified item assignments was submitted for confirmatory factor analysis (CFA) in MPLUS version 8 using robust maximum likelihood estimation and treating the item scores as continuous, consistent with guidance for Likert items with five or more categories (Rhemtulla et al. 2012). Overall fit of the model was evaluated using the root mean square error of approximation (RMSEA), the comparative fit index (CFI), and the Tucker–Lewis fit index (TLI).

For the purpose of summarizing results, correlation estimates of magnitude greater or equal to 0.40 and less than 0.60 are described as moderate, and correlation estimates of magnitude greater or equal to 0.60 are described as strong. Surveys with missing item-level data were excluded from calculations of item-total correlations and hierarchical omega for the corresponding subscale. All partially observed item data contributed to estimation for CFA, consistent with treatment of missing data in implementation of maximum likelihood estimation in MPLUS. The analysis plan included submission of the model corresponding to original item assignments for CFA and accommodation of clustering of responses by training date for the CFA; however, the CFA model associated with the original item assignments was not identifiable, and our data did not support fitting a model accommodating clustering. Item-total correlations and estimates of hierarchical omega were calculated using version 4.1.2 of R statistical software. Calculation of hierarchical omega used the MBESS package (Kelley 2007).

Results

Characteristics of the respondents

Two hundred sixty-two respondents accessed the pre-training survey, of whom 249 (95%) provided complete responses. Of the 13 (5%) respondents who did not complete all questions, 5 (2% of total) provided only partial or complete demographic information, and 8 (3% of total) provided responses for only some items, with a range of 1–39 of the 59 item ratings missing. Demographic characteristics of the respondents are summarized in Table 1. Two hundred fifty-eight (99%) reported postsecondary education, with 150 (57%) reporting a master’s degree or more education.

Table 1 Demographic characteristics of the n = 2621 survey respondents
Full size table

Assignment of items to subscales

Original and final modified assignment of items to subscales, magnitude of item-total correlations with assigned scales, and cross-correlations with alternate scales are provided in Fig. 2. Cross-correlations calculated using the original item assignments suggested that some reassignment of items to alternate subscales was appropriate. Items were reassigned in three stages, between which item-total correlations were recalculated and item performance re-assessed. In the first stage, two of the 25 items assigned to the self-assessed knowledge and attitudes subscale, “I care about whether my work is trauma-informed” and “Acknowledging cultural differences is an important component of a trauma-informed approach,” were reassigned to the system-wide knowledge and attitudes subscale based on weak correlation estimates (< 0.20) with the former and strong estimated cross-correlations with the latter. In addition, three items asking about “reflection on impact of cultural background on work” that did not correlate moderately or strongly with any subscale (magnitude of estimated correlations < 0.40) were assigned to their own subscale: “Awareness of cultural background at work.” This left only two items, “I am confident that my organization thoughtfully embraces and celebrates cultural differences” and “I am confident that my organization understands racialized trauma,” in the acceptance of cultural differences subscale. Because only two items remained in this subscale, and they cross-correlated strongly with the promotion of safety, acceptance and inclusion subscale, the items were reassigned.

Fig. 2
figure 2

Spearman item-total correlations (rhos) of the 59 scale items with their assigned subscales (+) and cross-correlations with alternate subscales, for the original and final item assignments to subscales. Items performing as expected have strong correlations with their assigned subscales and weak or no correlations with other subscales. TI, trauma-informed; TIC, trauma-informed care; TIP, trauma-informed practice; ACEs, adverse childhood experiences; MAMP, methamphetamine; SUD, substance use disorder; org, organization; HR, human resources; DEI, diversity, equity, and inclusion. I, self-assessed knowledge and attitudes subscale; II, safety and acceptance at work subscale; III, system-wide knowledge and attitudes subscale; IV, training and employee support subscale; V, promotion of safety, acceptance, and inclusion subscale; VI, interaction with clients’ subscale; VII, acceptance of cultural differences subscale; VIII, awareness of cultural background at work subscale. Prior to final item assignments, subscale VII was eliminated, subscale VIII was created, and subscales IV, V, and VII were combined into a single training, support, interaction, and environment subscale (IV/V/VI)

Full size image

Following the first stage of item reassignment, item-total correlations still reflected moderate or greater correlation of multiple items with more than one subscale for all but the self-assessed knowledge and attitudes, system-wide knowledge and attitudes, and awareness of cultural background at work subscales. Because all items on the promotion of safety, equity, and inclusion subscale correlated moderately with the training and employee support total score, and because both scores were thought to reflect trauma-informed practices at the systems level, items assigned to these two subscales were merged into a single subscale. For the final reassignment, because four of five items on the “interactions with clients” subscale correlated moderately with the merged subscale, and items on the interactions with clients’ subscale also rated trauma-informed practices of the system, its items were added to the merged system subscale, which was labeled the “training, support, interaction and environment” subscale. After this final reassignment, multiple items on the training, support, interaction, and environment subscale still correlated moderately with the safety and acceptance at work total score. However, items on these two subscales were not combined because items on the former were included to assess experience of trauma-informed practices on the individual level rather than to assess practices at the systems level. Estimates of internal consistency reliability as quantified by hierarchical omega for the original and final modified item assignments are presented in Table 2.

Table 2 Estimates of internal consistency reliability (95% confidence intervals) as quantified by hierarchical omega, original item assignments and final modified item assignments
Full size table

Factor structure

The RMSEA value for the CFA model corresponding to the final modified item assignments was 0.073 (90% CI 0.071, 0.076), a value associated with acceptable but not close fit. Neither the CFI value of 0.76 nor the TLI value of 0.75 approached the conventional threshold for acceptable fit of 0.90.

Standardized factor loadings for the CFA based on the final item reassignment ranged from 0.46 to 0.78 for self-assessed knowledge and attitudes, 0.59 to 0.87 for safety and acceptance at work, 0.76 to 0.93 for system-side knowledge and attitudes, 0.42 to 0.75 for training, support, interaction, and environment, and 0.75 to 0.97 for awareness of cultural background at work. All factor loadings were significantly different than zero.

Discussion

Our goal was to develop and validate a comprehensive assessment tool that measures trauma-informed, culturally responsive systems change across fields and systems. After development of the scale with a several step qualitative process, we obtained pre-survey measures from over 250 multi-disciplinary participants of a trauma-informed system change program. Following survey administration, analyses of the item-total correlations generally support expectations regarding the association of items with the underlying constructs measured by the scales. With the exception of five items that were reassigned to alternate subscales, items correlated moderately or more strongly with the total scores of their assigned subscales based on pre-specified item assignments. Estimates of internal consistency reliability quantified using hierarchical omega were also favorable after item reassignments, with lower limits of 95% CIs exceeding the common threshold of 0.80 for all subscales.

Low values for the CFI and TLI after final item reassignment likely reflect the correlation of some items with multiple underlying domains targeted by the scale. For the original item assignments, cross-correlations were particularly high among items intended to measure practices of the system and individuals within the system: training and employee support; promotion of safety, acceptance and inclusion, and interactions with clients. Combining items from these subscales reduced, but did not eliminate, moderate or greater cross-correlation. After the final item reassignment, items assigned to the knowledge and attitudes subscales and the awareness of cultural background at work subscales did not cross-correlate moderately or more strongly with any total score, but some moderately cross-correlating items remained for the safety and acceptance at work and training, support, interaction, and environment subscales. This cross-correlation of items for both the original and final modified item assignments could reflect the items’ association with a shared dimension underlying several of the domains targeted for measurement. For example, a workplace culture valuing diversity, inclusivity, and empathy may support adoption of a variety of trauma-informed practices at the systems level as well as a sense of safety and acceptance at the individual level.

Given that no alternative scale has been validated for evaluating trauma-informed knowledge, attitudes, and practices across industries and professions, this study demonstrates the STISC scale’s favorable overall performance. Furthermore, this study supports the use of the scale pending further study and refinement. Because of the high observed cross-correlations among items included to measure different aspects of trauma-informed workplace practices, we recommend scoring based on the final modified item assignments, which combine items corresponding to training and employee support, promotion of safety, acceptance, and inclusion by the system, and interaction with clients into a single subscale, rather than reporting totals for these workplace practices separately.

Study limitations and future directions

Limitations of the study include the small sample size relative to the number of items included on the scale, which prevented assessment of its factor structure using models of greater complexity and may have resulted in overestimation of the RMSEA and underestimation of the CFI and TLI (Shi et al. 2019). Items were not randomly ordered, and no items were reversed scored, which could make the scale susceptible to socially desirable response patterns. Respondents were highly educated, so results may not generalize to those without postsecondary education. Finally, the motivation for this work – the absence of a gold standard of measurement for trauma-informed practices in the general setting and research associating trauma-informed practices with characteristics of systems, knowledge, and attitudes – limited our ability to establish convergent and divergent validity.

Future studies investigating changes in scale scores following trauma-informed systems training and their predictors, and validating the scale using larger samples, will help to meet the need for measurement tools and understanding of the impact of trauma-informed practices. Further, in future studies, exploratory structural equation modeling using a larger sample of respondents can provide further insight into associations among the domains targeted by the scale and which items can best distinguish them. In addition, because modifications to item assignments were made based on the same data used to confirm the fit of the underlying model and estimate the internal consistency reliability of the subscales, results for the modified item assignments should be replicated using an independent sample. Lastly, sensitivity of the survey to change and degree of change in knowledge, attitudes, and practices following training will be addressed in a future study.

Implications and conclusion

Despite the need for further investigation using larger samples, our study demonstrates preliminary support for the STISC survey tool as a useful measure of trauma-informed practices and a cost-effective method of assessing trauma-informed systems change programs across multiple fields and industries. At 59 items, the Survey for Trauma-Informed Systems Change is brief, and easily administered and scored. Importantly, our survey is designed to be applicable in any setting. Trauma-informed care is becoming widely discussed in the media, government, and scientific literature (Becker-Blease 2017). Consequently, various systems have engaged with trauma-informed training absent cultural responsivity training and/or a standardized way to measure the impact of the training. Many existing validated scales measuring trauma-informed practices are limited in that they are designed for one specific system, such as education or health and human services, or they fail to adequately address the critical role of cultural responsivity in trauma-informed care. The goal of this study was to fill a gap by developing and validating a scale that can be administered to professionals in any field and will assess cultural responsivity at the individual and organization level. Our chosen subscales were based on SAMHSA’s key principles of a trauma-informed approach and their 10 implementation domains (SAMHSA 2014). The survey was hence designed to comprehensively capture the aspects of trauma-informed attitudes and practices that are predictive of positive outcomes for individuals, organizations, and systems, to include integrating knowledge about the consequences of trauma, fostering safety, trustworthiness, and transparency in the workplace through trauma-informed practices and services, and remaining sensitive to cultural, historical, and gender issues.

Evaluating the attitudes, beliefs, and practices surrounding trauma-informed care is becoming increasingly relevant as trauma remains a debilitating public health concern, pervasive across all systems. Thus, a standard way to measure trauma-informed care across all systems is needed. We echo the sentiment of previous scholars that critical to trauma-informed systems change efforts are psychometrically sound tools to measure the extent to which an organization or system is trauma-informed (Champine et al. 2019). Our hope is that the STISC can be used to help systems determine whether they are trauma-informed, to screen for and identify areas needing to be addressed and improved, and to guide trauma-informed intervention efforts and evaluate change in knowledge, attitudes, and practices in response to these efforts. As a next step in the validation process, we will need to determine whether our tool is sensitive to change. Further, pairing our survey results with outcome data that demonstrates meaningful change for stakeholders and those served will be important for showing that our tool is useful.

In sum, the Survey for Trauma-Informed Systems Change shows potential for providing the first psychometrically reliable and valid tool to help professionals from multiple systems, such as lawyers, judges, law enforcement officials, physicians, mental health care providers, educators, politicians, and many others, evaluate their culturally responsive trauma-informed practices.