Background

Overwhelmed healthcare workers struggle to find time for meals and bathroom breaks, let alone to initiate or fully engage in quality improvement projects. Yet, the safety and reliability of healthcare delivery depends on healthcare workers having sufficient capacity and support to engage in continuous self-reflection and quality improvement. How do leaders know if groups in their organization are capable of useful self-reflection, and ready to engage in meaningful quality improvement? To date, no validated scale exists to specifically measure Improvement Readiness (IR), or the ability of a work-setting to effectively participate in continuous learning.

Continuous learning around quality improvement can take many forms, such as integrating lessons learned from other work settings, incorporating insights and ideas of workers into the delivery of care, and learning from defects. The acts of self-reflection and learning are essential ingredients in all quality improvement, from introducing a new labeling procedure to launching (and subsequently fine-tuning) a system-wide electronic medical record. Entities and work settings vary considerably with respect to how well equipped they are to support continuous learning. In order to facilitate IR, many healthcare systems build organization-wide infrastructures to enable and support integrated assessments and feedback. These include explicit support from leadership and co-workers and protected time for continuous learning either on-the-job or via lectures and conferences [1,2,3]. Ideally, there are processes in place at the work-setting level to maximally learn, whether from deficits or other opportunities as they arise [4]. Conversely, key barriers to learning include time and financial constraints, lack of support from leadership and peers, and a negative workplace culture with insufficient processes to facilitate employees’ learning [1, 2, 5].

When IR is high, workers feel there are processes and norms within a work setting that encourage employees to learn, improve, and sustain quality efforts. Embedded within the infrastructure of the workplace environment, employees are empowered to recognize and address improvement opportunities while celebrating improvement efforts and positive results. Furthermore, areas with a high degree of IR are environments of psychological safety in which employees feel their opinions and suggestions are heard, that there is the capacity for them to be acted upon, and that improvements do occur [6,7,8].

Despite IR playing a vital role in healthcare quality, and even though the business literature lauds the importance of leaders who create IR, relatively little empirical attention has been paid to this construct in healthcare. Components of IR appear to somewhat overlap with constructs more often discussed in business, such as ‘organizational readiness for change’, ‘change readiness’, and ‘readiness for implementation’ [9,10,11]. A review of these constructs finds two overarching components: the motivational or attitudinal aspect of change (e.g., willingness) and the ability (e.g., being trained) [9]. The current conceptualization of IR was theoretically drawn from the ‘continuous learning and improvement’ component of Frankel and Leonard’s comprehensive approach to healthcare quality and safety [12]. It specifies that one’s ability to engage in continuous learning in one’s work setting is central, and even a precondition for, meaningful engagement in quality improvement. Although the IR construct overlaps with notions of ‘ability’ described in related work, scholars have not emphasized continuous learning and the conditions that support it. Despite a need for psychometrically robust scales for this construct, existing change or improvement-related measures suffer from limited evidence of reliability or validity [9]. The current study helps address this gap by making use of safety culture survey data collected across a large academic health system, which includes a novel measure of IR, five safety culture domains (e.g., teamwork and safety climate), and open-ended text box responses.

Safety culture surveys often end with open-ended text boxes for optional comments, however, no study, to our knowledge, has used the text from these types of comments in an empirical article to examine linguistic differences across various groups in healthcare. Linguistic research has found that underlying psychological factors appear to express themselves through word choice [13,14,15,16]. Based on prior research, we expected that healthcare workers who work in settings with low IR are likely to reveal their lack of resources and higher stress, through a less healthy pattern of language. Therefore, we expected workers from low IR work settings to more frequently use first person singular pronouns [13, 17, 18], words that focus on the past (i.e., past verb tense, indicative of being in reactive mode) [19,20,21], and negative emotions words [14], and to less frequently use first person plural pronouns (i.e., “we/our/us”) and positive emotions [14, 22], and social words (e.g., “team”) [23] compared to high IR work settings. Finally, since open-ended text prompts may be one of the few opportunities for staff members from under-resourced or stressed work setting to speak up about concerning issues, we expected workers from low IR work settings to submit longer responses (i.e., more words).

The current study had four aims which each served to clarify the nature and measurement of IR. The first aim was to examine the psychometric properties of a novel IR scale. The second aim was to assess relationships between IR and other safety culture measures from SCORE [7]. We expected IR to correlate positively with Local Leadership, Safety Climate, Teamwork Climate, and to correlate negatively with Personal Burnout, Burnout Climate, and problems with Work-life Climate [7]. The third aim was to explore whether IR differs by healthcare worker role, shift, shift length, and years in specialty. The fourth aim was to examine linguistic differences in word type use between high and low scoring IR work settings in their free text responses.

Methods

Design and study population

This is a cross sectional study of 2016 survey data sent to 13,040 healthcare workers across 440 work settings within one academic health system as part of the Safety, Communication, Operational, Reliability, and Engagement (SCORE) survey [7]. This study was approved by the Duke University Health System Institutional Review Board.

All staff with 50% or greater full-time equivalent commitment to a specific work-setting for at least four consecutive weeks were asked to participate. Work settings with five or more respondents and a response rate of at least 40% were included in the aggregated analyses (i.e., domain level correlations), resulting in a sample of 396 work settings (90%).

Measurement of improvement readiness (IR)

Improvement Readiness measure is a sub-scale within the SCORE survey, which can be used on its own or as a part of SCORE [7]. The scale’s five items were presented to the respondent as phrases that follow an initial prompt that says: “The learning environment in this work setting …”, e.g., “Integrates lessons learned from other work settings.” Respondents are asked to rate the items on a 1–5 scale, (1 = disagree strongly, 5 = agree strongly). See Fig. 1.

Fig. 1
figure 1

Standardized Factor Loadings for Improvement Readiness Scale

Full size image

SCORE survey

Scales assessing Work-life Climate, Teamwork Climate, Safety Culture, Burnout Climate, Personal Burnout, and Local Leadership, were also measured within the SCORE survey [7].

Statistical analysis

Improvement Readiness scale scores were calculated by taking the mean of the five items for each respondent using the 5-point scale. To assess the extent respondents perceived IR favorably across their work settings, or in other words the IR climate, an aggregate percentage of positive responses was computed by calculating the percentage of respondents in each work setting who scored greater than or equal to a 4 (e.g., those who, on average, “agreed slightly” or “agreed strongly”). IR aggregated climate scores were only used in the current study to examine associations with other safety culture scores, which were also computed using the same technique.

Confirmatory factor analysis (CFA) with maximum likelihood estimation was used to examine the fit of a one-factor model for the 5 IR item scale. Fit was assessed using the following indices: Root Mean Square Error Approximation (RMSEA; < .08 considered adequate fit) [24], Confirmatory Fit Index (CFI), and Tucker-Lewis Fit Index (TLI; CFI and TLI’s > .95 considered acceptable fit) [25]. Internal reliability was assessed using Cronbach’s alpha (α; > .70 is considered acceptable) [26]. Analysis of variance (ANOVA) was used to test for differences on the IR scale by healthcare worker role, shift, shift length, and years in specialty. A Random Effects ANOVA tested both within work-setting variance and between work setting variance for the IR scale. An Intraclass Correlation Coefficient (ICC) was computed to assess the extent of dependence or clustering of IR scores based on work-setting, which supports the use of aggregating scores at the work setting level [27]. Statistical analyses were performed using IBM SPSS V.24 [28] and Mplus version 7.4 [29].

Linguistic inquiry and word count analysis of comments

Linguistic word count patterns were assessed with the Linguistic Inquiry and Word Count (LIWC) software program, version 2015 [30]. LIWC uses an internal dictionary of 6400 words and word stems to process text files to calculate the percentages of total words that fit into 80 language categories. The categories include standard linguistic dimensions, social and psychological processes, affect-related words, and non-psychological processes [31]. The word counts are expressed as a percentage of the total number of words used thus controlling for the length of the input text file.

The current study hypothesized different calculated word percentages in specific pre-determined linguistic categories for open-ended comments from work settings coming from the highest versus lowest 10 % across the current sample in Improvement Readiness. Top and bottom deciles were selected to facilitate thematic analysis of these comments for a separate project, ensuring that every comment subjected to LIWC in the current study was also read by multiple people. The SCORE survey included two open ended questions: “Please share something you have seen make a positive impact on the culture in your work area that you recommend continue”, and “Do you have any other comments, questions, or concerns?”. To maximize the amount of text examined for each hypothesis, the responses to both questions were combined for each respondent.

To visualize the frequency of words used, Wordles were created for the high and low IR work settings. Wordles are word clouds that display the size of a word proportionally to the number of times the word appears in a section of text.

T-tests were used to compare linguistic category count variables for the high and low IR work settings. For T-tests that failed Levene’s test for equality of variances, we report values that did not assume equal variances.

Results

Respondent demographics

Electronic surveys were returned by 10,627 out of 13,040 possible survey respondents (overall response rate 81%). Table 1 presents demographic data for respondents.

Table 1 Respondent Demographics and Improvement Readiness Cronbach’s α
Full size table

The top three respondent groups were registered nurses (31.7%; n = 3367), attending physicians (9.7%; n = 1036), and technologists (8.2%; n = 869). A subset of respondents (3.2%) did not identify with one of the listed healthcare worker roles. Respondents were predominantly day-shift workers (68.1%), with diversity in years of experience in their specialty and shift length. Missing data for each of the items ranged from 0.9 to 3.2%.

Aim 1) psychometrics and confirmatory factor analysis

The mean IR score across those who completed the scale (n = 10,154) was 4.08 (SD = .97). The mean percent positive IR climate score across the 396 work settings was 77.28 (SD = 12.74) and work settings ranged from 38.91 to 100% positive IR climate (See Fig. 2). The overall sample’s Cronbach’s alpha, an index of internal reliability, was .93. Above .70 is considered acceptable, particularly for shorter scales [26]. The alpha ranged from .80 to .96 across various demographic groups (see Table 1).

Fig. 2
figure 2

Percent of Respondents Reporting Good IR across 396 Work Settings

Full size image

A one factor model of the scale produced good fit to the observed data (RMSEA = .07, 90% CI: .06–.078; CFI = .99; TLI = .99). Standardized factor loadings ranged from .79 to.86 (See Fig. 1).

A Random Effects ANOVA of the entire sample revealed significant variance within (.847) and between (.093) work-setting variance in IR scores, p-values < .001. The computed ICC was 0.10, indicating 10% of the variance in IR scores can be attributed to between work-setting characteristics. In other words, there was significant clustering of IR responses within work settings, reflecting a non-trivial level of consensus at the work setting level.

Aim 2) associations with safety culture and burnout scales

IR significantly predicted all other Safety Culture scales in expected directions. Specifically, IR was positively correlated with Local Leadership, Safety Climate, and Teamwork Climate. IR was negatively correlated with Personal Burnout, Burnout Climate and problems with Work-life Climate (See Table 2). Correlation coefficients ranged from .756 (Safety Climate) to .405 (Work-life Climate).

Table 2 Correlation matrix for Improvement Readiness and additional healthcare climates surveyed. Cronbach’s alpha for each domain included in the diagonal
Full size table

Aim 3) IR by role, shift, shift length, and years in specialty

Univariate ANOVA demonstrated significant differences in the IR scale between healthcare worker role (F(18, 10,135) = 12.92, p <  0.001), shift (F(4, 10,154) = 11.94, p <  0.001), shift length (F(5, 10,155) = 10.27, p <  0.001), and years in specialty (F(7, 10,155) = 13.54, p <  0.001).

Scheffé post hoc tests revealed Administrators/Managers/Supervisors reported significantly better IR than 13 of the 18 possible roles. Administrators/Managers/Supervisors reported statistically equivalent IR to Clinical Support Workers (e.g., Medical Assistant, EMT), Dietician/Nutritionists, Physician Fellows, and Environmental Services. Psychologists and Clinical Social Workers/Case Managers reported significantly lower IR than all other possible roles in the current sample.

Scheffé post hoc tests revealed day shift workers reported higher IR than night and ‘other’ shift workers. Swing workers reported higher IR than ‘other’ shift workers. Eight-hour shift workers reported higher IR than 10-, and 12-h shift workers, and those who described their shift as ‘other’.

Workers with fewer years in specialty generally report higher IR. Specifically, those less than 6 months in their specialty reported significantly higher IR than every other length in specialty, except for those 6–11 months in specialty.

Aim 4) linguistic analyses

Percent positive IR scores (percent that on average, agreed with IR items) were computed for each work-setting. The top and bottom 10% of work settings were identified. The work settings in the lowest 10% (39 work settings) had a mean IR score of 52.51 (SD = 6.54). The work settings in the highest 10% (39 work settings) had a mean IR score of 97.15 (SD = 1.92).

Three hundred and fifty-five workers provided comments from the low IR work settings (31.4% of the 1129 who completed the survey from these groups). One hundred and forty-nine workers provided comments from high IR work settings (27.3% of the 536 who completed the survey from these groups). Wordles, which graphically depict word frequency, were created for the high and low IR groups, and can be found in Fig. 3. Representative comments from high and low IR work settings can be found in Table 3.

Fig. 3
figure 3

Wordles of Comments from Low (left) and High (right) IR Work Settings

Full size image
Table 3 Representative Comments from the Top and Bottom 10% IR groups
Full size table

Table 4 and Fig. 4 show the linguistic analyses results. As hypothesized, workers from low IR work settings used significantly more words overall in their responses, as well as more past tense verbs, first person singular, and negative emotions words, compared to those from high IR work settings. As a follow-up to the negative emotion words result, exploratory analysis found the rates of the three negative emotion sub categories, anxiety, anger, and sadness, were significantly greater for low IR work settings than for high IR work settings.

Table 4 T-test results for LIWC analyses comparing high and low IR work settings
Full size table
Fig. 4
figure 4

Word Counts for High and Low IR Work Settings

Full size image

As hypothesized, workers from high IR work-settings used significantly more positive emotions words and social words. Wordles appear to reflect the findings from the linguistic analysis. For example, in Fig. 3, note the large size of the words, “together,” “team,” and “help” reflect that high IR work settings use more social words.

Discussion

In this study we provide evidence for the psychometric soundness, reliability, and convergent and construct validity of a novel IR scale.

Aim 1) examining the psychometric properties of a novel IR scale

The IR scale meets and exceeds established psychometric thresholds for reliability [26], and the single factor model provided good fit to the data, demonstrating construct validity [24, 25, 32]. Moreover, the IR construct appears to behave like a group norm or climate, as evidenced by the non-trivial clustering within a work setting.

Aim 2) assessing relationships between IR and other safety culture scales

The IR scale was clearly associated with established safety culture and burnout scales in expected directions. Specifically, work settings with higher IR also report better Local Leadership, Safety Climate, Teamwork Climate, and Work-life Climate, as well as lower Personal Burnout and Burnout Climate. Work settings that are struggling with issues related to leadership, patient safety and teamwork are not yet at a place where they can meaningfully engage in learning to improve quality. Rather, and understandably, their efforts are focused on reacting to current deficits. The IR scale offers a window to managers and leaders as to which groups are best prepared to trial a new initiative or process to improve quality.

Of particular note is that the highest loading IR item was “The learning environment in this work setting allows us to gain important insights into what we do well.” This suggests that pausing and reflecting on what is going well is a key component of IR, which is interesting considering that improvement efforts often focus exclusively on what is not going well. The importance of positive reflection to gain insight was captured in the positive emotions linguistic finding, as well as in several comments from the top 10% IR group, e.g., “[My leader] does a fantastic job at checking in on employees to see what is going well, what can be improved, any concerns that need to be addressed. She follows up on each of these things. She writes a card to those who are recommended for recognition and hand delivers them! She is the best Manager I have ever had and feel fortunate to be under her. She is a rock in this department!!!!”.

Aim 3) exploring whether IR differs by healthcare worker demographic factors

Roles, shift type, shift length, and years in specialty differentially predicted rates of IR. In general, those with fewer years in specialty, who work day shifts, and are 8-h shift workers, reported higher IR. If poor IR is to a group as poor work-life balance is to individuals, then it might represent available bandwidth to learn and participate in quality improvement. To that end, it isn’t surprising that many of the differences in IR by demographics look similar to their counterpart differences in poor work-life balance [33]. Future research would benefit from understanding what factors play a role for those reporting higher IR, so that these factors could be adopted more broadly. For example, it may be that those much earlier in their careers receive more support for their learning and development, that they receive more feedback on their progress, or that they are simply less burned out than those farther along.

Larger work settings generally have lower IR scores than smaller work settings. Specifically, the number of people in the lowest IR decile of work settings was twice as many as the number of people in the highest decile. Perhaps it is more difficult to coordinate improvements in larger groups. Larger groups are also more likely to have higher worker to management ratios and higher acuity patients, and therefore are less likely to have effective feedback and quality improvement discussions, which are components of IR.

Aim 4) examining linguistic differences in word type use between high and low scoring IR work settings in their free text responses

Linguistically, we learned higher performing IR work settings speak less in the first person singular (I/me/my), and more in the first-person plural (we/our/us; at the level of a trend). This mirrors the established finding in linguistics research that greater first person singular use is a marker of depression, since emotional pain tends to draw attention to people’s own experiences [13, 14, 27]. In low IR work settings, this may reflect lacking control over one’s environment, leading to feelings of, anger, sadness, and anxiety, which were all word categories that were higher in low IR settings. High IR work settings used less past tense, indicating they are less likely to be in reactive mode, addressing prior deficits [19, 20]. As anticipated, high IR settings also used more social words (a marker of wellbeing [33]), and reinforced by the 44% of the variance shared between IR and Teamwork Climate. Finally, it is particularly noteworthy that high IR work settings used more positive emotion words, given that psychological research consistently shows that better mental health and wellness is associated with more positive emotion word use [34,35,36].

Significant differences in word use in the top and bottom IR deciles suggest, in combination with the ICC results and the strong correlations with personal burnout, that there is a climate of IR. In other words, IR appears to operate as a group norm, that reflects more well-being and ability to take on new initiatives. The ICCs suggest this through the clustering of IR scores within work settings, the burnout correlations suggest this is related to wellness, and the LIWC results provide additional linguistic support.

This study is limited in its use of self-report data which are at risk for response, selection, and social desirability biases. This was a study of one large academic health system, thus, findings may not generalize broadly to other healthcare systems, or outside of healthcare. The linguistic findings from this study are limited by using only the top and bottom deciles, which were selected to facilitate thematic analysis of these comments for a separate project, ensuring that every comment subjected to LIWC in the current study was also read by multiple people. We therefore excluded 80% of the potential comments from these analyses. Nevertheless, 504 individuals’ comments, which totaled 33,076 words, were analyzed in the current work. Although there were 80 LIWC categories that could have been analyzed, we limited our selection to the most theoretically relevant categories. We merged responses to two distinct open-ended question prompts in order to maximize the number of words available for each respondent. However, we did look at the comments for each prompt separately and found a similar pattern of results. Finally, we did not assess whether work settings higher on IR actually achieve greater quality improvement success, an area for future research.

Conclusions

Successful quality improvement initiatives hinge on the readiness of work settings to take them on. People interested in assessing IR have strong support from this study for the use of the brief, interpretable and reliable IR scale. The IR scale exhibits robust psychometric properties, is associated with established safety culture scales in expected directions (i.e., better safety and teamwork climate, lower burnout), and predicts linguistic differences in high versus low IR groups. As part of a safety culture assessment, IR provides a novel insight into the ability of a group to improve, which is distinct from, e.g., teamwork norms or patient safety deficits. The IR scale can help determine, among work settings with otherwise similar safety culture profiles, where we can expect readiness for improvement and where we cannot.