Measuring the Capacity of Staff Culture to Further Recovery from Psychiatric Disabilities
- First Online:
- Cite this article as:
- Chandler, D. & Wilson, S. J Behav Health Serv Res (2014) 41: 153. doi:10.1007/s11414-013-9348-9
- 1.3k Downloads
Three linked instruments for measuring the recovery-orientation of mental health program culture are introduced as the Recovery Centered Measures (RCM). Two scales assess the views of staff and of consumers, respectively, regarding staff–consumer interactions. A third scale measures staff culture. The RCM scales are quick, easy to understand (reading level of grade 5.4), and internally consistent. Test–retest correlations ranged from 0.81 to 0.67. Convergent validity with three related instruments was appropriate. The scales discriminate ACT from residential programs. The RCM scales show strong potential to be useful to program administrators and researchers working to increase the recovery-orientation of programs.
This paper focuses on the process by which persons recover those parts of the self and life that have been compromised by participation in mental health treatment systems and the stigmatizing culture in which treatment is embedded.1 In the development of a psychiatric disability, individuals experience a series of losses, including loss of hope, personal power, uniqueness, motivation, identity, self-respect, self-acceptance, relationships, and the opportunity to learn skills for making effective choices.2 While some losses are best addressed by individuals or with individual interventions—such as awakening hope, exploring identity, enhancing connections with others, reducing harm, and learning the skills of making effective choices—others are better addressed by seeking to increase the recovery-orientation of treatment program culture. The culture, and the interpersonal interactions that comprise it, become a primary intervention facilitating individuals’ recovery from serious mental illnesses.
During the era when state mental hospitals and therapeutic communities predominated, research often focused on ward culture,3 and this anthropological approach was extended to the early PACT program in Madison.4 In a National Association of State Mental Health Program Directors study investigating what helps and what hinders recovery, Onkin and colleagues extend this cultural approach by focusing on the role of the “organizational culture” of formal mental health services.1
This article presents and tests the Recovery Centered Measures (RCM scales), which have been developed to measure the recovery-orientation of cultural beliefs, values, norms, and practices in treatment programs as they are expressed in interactions between staff and consumers and between staff with other staff.
The scales measure how much programs vary in the extent to which staff manifest respect for persons-served, recognize their individual uniqueness, increase their motivation, accept them non-judgmentally, and share power with them. These five dimensions come directly from Onkin and colleagues, but are echoed many places in the literature.1, 2, 5, 6, 7
The RCM scales are designed to be used to measure change in program culture and to compare the recovery-orientation of different programs. They are also used to focus culture improvement initiatives. The three scales have different viewpoints: staff views of their interactions with persons-served (RCM-Staff View), consumer views of their interactions with staff (RCM-Consumer View), and staff views of their interactions with other staff (RCM-Staff View of Staff). The two studies presented here describe the psychometric properties of all three RCM scales.
To our knowledge, no other instruments specifically focus on staff views of their own culture, although there is a large literature on treatment milieus from which one can infer the importance of staff interactions with each other. For example, the Ward Atmosphere Scale8 includes items, responded to by both staff and clients, like this: “Staff sometimes argue openly with each other.” Our attention to the importance of the intra-staff dimension was triggered by receiving a letter from a patient in 2002 that included this statement: “In 35 years in the Mental Health desert, probably the greatest desert of all Mankind, this is the 1st place I have been where the workers care about the Happiness of their fellow workers and the Happiness of the Patients.” Some empirical confirmation is found in a study of inpatient units that found a strong correlation between patient satisfaction on one hand and staff satisfaction and staff working conditions on the other.9 Likewise, psychiatric inpatient seclusion rates have been found to depend partially on such factors as trust between staff.10
Based on literature analysis by the RCM scale developers, a pool of 235 items was generated for the three versions of the scale. The items were organized by the five components of recovery described above, with an average of 11.5 items for each concept in each scale. To the extent possible, the three scales were designed to be parallel. The scale in which staff assess culture among themselves is necessarily somewhat different in concept and wording.
Recovery centered measure item meansa and item correlations with the test (from study 2)b
RCM-Consumer View itemsc by subscaled
RCM-Consumer View mean (standard error)
RCM-Staff View mean (standard error)
RCM-Staff View of Staff mean (standard error)
RCM-Consumer View correlation with the test
RCM-Staff View correlation with the test
RCM-Staff View of Staff correlation with the test
Staff members accept me for who I am.
Staff members are interested in my background.
Staff members are interested in my spiritual or religious beliefs.
Staff members work with me to identify my strengths.
Staff members don’t really listen to me.
Staff members make judgmental statements about clients.
I can talk to staff members without feeling judged.
Staff members listen to me rather than telling me what I ought to do.
Staff members work with me without judging me.
Staff members gossip about clients.
Staff members motivate me to improve my life.
Staff members give me hope for a better future.
Staff members help me feel committed to improving my life.
Staff members inspire me to pursue my recovery.
Staff members don’t work with me to realize my hopes and dreams.
Staff members make me feel more effective in my life.
Staff members don’t consult me when making decisions that affect me.
Staff members work in partnership with me.
Staff members encourage me to make my own choices about my life.
Staff members make me feel less powerful.
I see staff members being disrespectful to other clients.
Staff members treat me as a valued person
Staff members don’t believe what I have to say is important.
The way staff members treat me increases my personal dignity.
Staff members show me respect when I make efforts to improve my life.
The first study reported here included one large facility-based and five ACT programs; in study 2 there were two facility-based and two ACT programs. Study 2 differed from study 1 in two ways: (a) one item that had a low correlation to other items in study 1 was replaced and a number of small wording changes were introduced, and (b) one of the instruments used for convergent validity in facility-based programs in study 1 was replaced.
During two pretests, developers had to resolve a number of questions about scale administration for persons-served including how to distribute and get back the test forms (more of an issue in ACT programs) and how to deal with low literacy, vision problems, or cognitive limitation. During the pretests, it was also decided to include indirect staff and administrators as well as treatment staff in the RCM term “staff.” An administration manual is available from the authors.
Instruments for concurrent validity
Two recent reviews11, 12 identify six instruments designed to capture the perspectives of persons-served regarding the recovery-orientation of the care being provided. For determining convergent validity, the two scales among the six that are most focused on program culture and have some tested psychometric properties were chosen. The Recovery Enhancing Environment (REE) instrument was developed by Patricia Ridgway with much consumer input. It has been field-tested in two studies, but only Cronbach’s alpha has been reported.13, 14 There are several scales included in the instrument; the 14-item Organizational Climate scale is used here. Although originally intended to be answered by consumers, the items can be answered by staff with no translation or modification.
The Recovery Self-Assessment scale is intended to measure the extent to which programs implement recovery-oriented practices. The Recovery Self-Assessment scale includes versions for staff, families, administrators, and consumers. Although a field test of the instrument was published by the authors, the only psychometric data from the study is an exploratory factor analysis.15 A replication study of only the staff version showed high internal consistency and, in a sample of 34 staff members, a test–retest intraclass correlation of 0.83.16 The Recovery Self-Assessment has been used to test staff change after a year-long training effort on recovery.17 In a study of 67 Canadian ACT programs, the Recovery Self-Assessment was used to assess the relationship between recovery-orientation and outcomes and recovery-orientation and ACT fidelity.18, 19
A third instrument is the WARD Atmosphere Scale, a widely used scale developed by Rudolph Moos in the 1970s to measure organizational and cultural features of therapeutic communities. It is supported by extensive psychometric data.8
Psychometrics of instruments used to test convergent validity (as determined in this study)
REE Organizational Climate scale
Recovery Self-Assessment scale
Ward Atmosphere Scale
Test–retest intraclass correlations
REE Organizational Climate scale
Recovery Self-Assessment scale
Ward Atmosphere Scale
The primary goals in selecting programs to participate were to include both facility-based and ACT programs and to balance numbers of staff with numbers of persons-served. All the programs in both studies are operated by a large California provider. The first study included 201 staff and 201 persons-served; the second study included 220 staff and 227 persons-served. In large programs, persons-served were sampled randomly to obtain a study group roughly equal in size to the number of staff. The study refusal rate for persons-served was 9.1 % in study 2 (but not recorded consistently for study 1). Persons-served who were judged by staff to be too impaired by symptoms to complete the study forms were excluded as were persons with limited English language capability. (At this time the RCM forms are available only in English.) A method was used to permit linking of test and retest forms to the same persons while still preserving anonymity.
Characteristics of staff
Residential N = 109+ (%)
ACT N = 92 (%)
Residential N = 162 (%)
ACT N = 56 (%)
56 and over
Time working in mental health programs
Less than 5 years
5 to 10 years
Over 10 years
Role in program
Direct service paraprofessional
Direct service professional
Characteristics of persons-served
Residential N = 92 (%)
ACT N = 102 (%)
Residential N = 86 (%)
ACT N = 141 (%)
56 and over
Self-reported psychiatric and substance abuse
Time receiving mental health services
Less than 1 year
1 to 5 years
5 to 10 years
More than 10 years
Performance of the RCM in study 1 and study 2a
Minutes to complete: RCM-Consumer Viewb
6.38 (SD = 4.65)
14.18 (SD = 15.20)
Minutes to complete: combined RCM-Staff View and RCM-Staff View of Staff
8.44 (SD = 3.81)
12.87 (SD = 8.85)
Percent missing data: persons-served (RCM-Consumer View)
Percent missing data: combined RCM-Staff View and RCM-Staff View of Staff
RCM-Staff View scale Cronbach’s alpha
RCM-Staff View of Staff scale Cronbach’s alpha
RCM-Consumer View scale Cronbach’s alpha
Test–retest: RCM-Staff View of Staff intraclass correlation
Test–retest: RCM-Staff View intraclass correlation
Test–retest: RCM-Consumer View intraclass correlation
Correlation of RCM-Staff View and REE Organizational Climate scale as completed by staff
Correlation of RCM-Staff View and RSA staff scale
Correlation of RCM-Consumer View and REE Organizational climate scale completed by consumers
Correlation of RCM-Consumer View and RSA consumer scale
Correlation of RCM-Staff View of Staff and REE Organizational Climate scale as completed by staff
Correlation of RCM-Staff View of Staff and RSA staff scale
Discriminant validity: RCM-Consumer View
Discriminant validity: RCM-Staff View scale
Correlational and subscale structure
Each RCM scale contains 25 items which are hypothesized to represent an underlying latent dimension of recovery-oriented culture. Factor analysis on the items in each scale was conducted using the principal factors method. The number of retained unrotated factors was determined using a screeplot. For the two staff-completed scales, in both study 1 and study 2, there is one predominant factor and one subsidiary factor (eigenvalue 2.0 or less) that loads predominantly or entirely on the reversed items. On the persons-served scale, in study 1 there is one predominant factor and a second factor (eigenvalue 3.23) which loads entirely on the reversed items; in study 2 there is one predominant factor and one secondary factor (eigenvalue 3.99) which again only loads on the reversed items.
By design, the RCM contains five subscales of five items each: individual uniqueness, non-judgmentalism, shared power, motivation, and respect. The overall scale was developed to measure these five conceptual aspects of recovery culture. An exploratory factor analysis of each subscale was conducted using the principal factors method and found no more than one factor (eigenvalue of 1.0 or more) in each subscale. Items also correlated higher with their own subscale than with any other subscale (out of all 75 correlations of each item with five subscales in three scales, there were five exceptions to this finding in study 1 and 3 exceptions in study 2). However, the correlations between subscales of each RCM scale are relatively high: RCM-Consumer View correlations range from 0.69 to 0.79; for the RCM-Staff View, the range is 0.63 to 0.83; and for the RCM-Staff View of Staff, the range is 0.54 to 0.73. This amount of correlation indicates the single underlying latent variable is a strong component of each subscale.
Internal consistency is measured with Cronbach’s alpha. The staff view of staff RCM scale has an alpha of 0.93, with subscales ranging from 0.71 to 0.84. The staff view of interactions with consumers RCM scale has an alpha of 0.95, with subscales ranging from 0.74 to 0.88. The consumer RCM rating scale had an alpha of 0.93, with subscales ranging from 0.55 to 0.81 (see Table 5). Thus, internal consistency is high for the scales and acceptable to good for the subscales.
In study 2, the three RCM scales were administered twice. For the consumer RCM scale there were 96 participants with time between test and retest averaging 13.4 days (range 7–21). The test–retest intraclass correlation was 0.67. For the two staff RCM scales, 128 participants averaged 14.1 days between administrations (range 7–20). The intraclass correlation of test with retest for the staff view of consumer scale was 0.72; for the staff-view-of-staff scale, it was 0.81 (see Table 5). These test–retest correlations are also acceptable to good.
Several approaches suggest the RCM scales are valid measures of program culture in relationship to furthering or hindering recovery of persons-served. The RCM items and subscales have face, content, and construct validity due to their genesis in consumer writings and research about recovery and to the use of consumers and line staff in refining and selecting items. RCM convergent validity was judged by comparing RCM scores to those of the three related instruments described above. Correlations between the Ward Atmosphere Scale and the RCM scales range between 0.30 and 0.57. The REE Organizational Climate scale correlates with the staff-view and consumer-view RCM scales between 0.56 and 0.71. The third convergent validity instrument is the broader 32-item Recovery Self-Assessment scale, which has a consumer and a staff version. Correlations between the RCM scales and the Recovery Self-Assessment scale ranged between 0.39 and 0.72. Please see Table 5. These correlations indicate substantial sharing of content but not redundancy.
Another measure of validity is the concordance between ratings made by staff and by persons-served. Since the presumption of the study is that there is an underlying latent variable of “program culture which does or does not support recovery”, it is hypothesized staff and persons-served will rate the same items high or low and to the same degree. Correlation between staff and consumer means when rating the same items is high (Pearson’s r = 0.78). However, agreement is reduced by the fact that staff consistently rate items somewhat higher than persons-served (bias). The product of the correlation coefficient and a measure of the bias is summarized in the correlation of concordance, which for the RCM is 0.65.20
Finally, validity is confirmed by the finding that the RCM scales discriminate between program types. Scores for both consumer and staff RCM scales were expected to be lower for facility-based programs than in ACT programs, and, as shown in Table 5, scores are lower in facilities in both studies.
There is considerable interest in the behavioral health community in measuring the recovery orientation of programs.11, 12 It is useful to compare the RCM scales with the two instruments these authors reviewed which most focus on program culture. These are the same instruments used for testing convergent validity: the REE Organizational Climate scale and the Recovery Self-Assessment scale (RSA).
Correlations of the RCM scales with these related scales are substantial, 0.30 to 0.72, but not high enough to suggest the RCM scales measure the same latent dimension. By way of comparison, Andresen and colleagues recently looked at four individual recovery self-rating scales and found correlations ranging between 0.40 and 0.89.21
The REE Organizational Climate scale and the RCM scales are relatively quick to administer; the RSA is somewhat longer (A very short consumer version has recently been developed but 205 psychometrics have not been published.) All three were developed with considerable consumer input. The RCM has a reading level of Flesch–Kincaid 5.4; the REE reading age is 12–13 and that of the RSA 15–16.12
The Recovery Self-Assessment and the RCM scales have both consumer and staff versions. The RSA additionally has an administrator and family version while the RCM has a version measuring staff views of their own culture. The REE Organizational Climate scale was developed for consumer rating, but data presented here shows that it has good psychometric properties when rated by staff as well.
The content of the Recovery Self-Assessment scale is considerably broader than program culture. For example, items include “The physical space of this program (e.g., the lobby, waiting rooms, etc.) feels inviting and dignified” and “I am encouraged to attend agency advisory boards and/or management meetings if I want.” The RCM focuses specifically on concepts important to recovery as they manifest themselves in staff–consumer and staff–staff interactions. Most REE Organizational Climate items also focus on interactional culture, but there are items on physical appearance and program resources and consumer satisfaction as well.
In summary, if program or organizational culture is the interest, the RCM scales are the most focused, with the REE Organizational Climate and the Recovery Self-Assessment scales following in that order. The RCM scales and subscales have more conceptual clarity than the other two scales; the RCM subscales also have good validity and internal reliability but cannot be interpreted as underlying causal dimensions as would be the case if they were defined by confirmatory factor analysis (the RSA factors have also not been confirmed). Psychometrically, all the scales are acceptable but psychometric testing of the RCM scales is most complete to date. The only instrument with an administrator and family version is the Recovery Self-Assessment; the only instrument that measures staff interactions with other staff is the RCM.
The two RCM studies are limited in several ways. First, instruments used for convergent validity in the studies are not entirely apposite; it is particularly difficult to find appropriate comparisons for the instrument measuring staff view of interactions with other staff. Second, all the programs included in the studies are operated by one agency, so it will be important to test the RCM scales in other contexts. Third, the item order on the RCM scales was assigned randomly, but this may have increased the reversed-item bias as several reversed items were clustered together. Fourth, all the programs in these studies serve persons with very serious functional difficulties (ACT or residential rehabilitation facilities), so it is not yet known if the RCM scales will prove useful in peer-support programs oriented to wellness and recovery. Fifth, the RCM scales have not been tested in other languages or cross-culturally. And finally, it remains to be proven that higher RCM scale scores for programs are associated with better individual recovery outcomes.
Implications for Behavioral Health
Program culture has a profound and pervasive effect on the recovery of individuals served by a program. Changing the culture of a program to better support recovery can be a principal intervention practitioners use to assist the persons they serve. The RCM measures can be employed in several ways to further this effort.
An initial step is to administer all three scales to a program and provide feedback of scores to staff using graphs. The feedback is designed to help motivate improvement by identifying domains, defined by subscales scores, where the program is doing less well and by pointing out specific discrepancies between staff and consumer ratings. For example, in the studies reported here staff learned that they rate their respect for the spiritual beliefs and practices of persons they serve far higher than do those persons themselves.
In addition to its use in individual program cultural transformation efforts, the RCM scales can, like the more broadly focused RSA, be used to profile multiple programs15 or to serve as an intervening or mediator variable in research on attainment of individual recovery goals.22
The imperative for programs to assist persons-served in their journeys to recovery is no longer a matter of debate. The conversation has shifted to how programs can best accomplish this goal. The RCM scales show strong promise as tools for helping programs identify what aspects of organizational culture need to change in order to promote recovery. Uniquely among related instruments they have strong psychometrics, focus solely on interactional culture, and include a scale for measuring the recovery orientation of staff-with-staff interactions.
Others instrumental in developing the RCM scales are Stacey Calhoun, Joan Meisel, and Jesse Tamplen. Marilyn Schlagel managed administration of the study instruments.
Conflict of Interest
Daniel Chandler is an independent consultant contracted by Telecare Corporation to conduct research refining and testing the RCM scales. Stephen Wilson is medical director for Telecare Corporation and developed the initial versions of the scales. Neither the authors nor Telecare Corporation will receive compensation for use of the RCM scales by other parties. The scales are accessible at http://www.telecarecorp.com/page3-61/RCMAvailability. Authors declare no other conflicts of interest.
Open Access This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.