Abstract
The concept of recovery can be operationalized from either the point of view of the consumer or from the perspective of the provider of services. The Recovery Markers Inventory (RMI) was created to assess recovery-related factors (i.e., actions/events associated with consumer’s recovery) from the provider’s perspective. Evidence, which established the psychometric properties of the RMI, was obtained through the use of: (a) construct validity (i.e., confirmatory factor analysis and Rasch principal components analysis of residuals); (b) concurrent validity (i.e., the calculation of Pearson correlations between the RMI and other recovery-oriented instruments); and (c) reliability (i.e., Rasch Partial Credit models). Evidence presented in this article shows that the RMI scale is unidimensional, has an adequate level of correlation, and acceptable reliability. The current analysis provides evidence to support the RMI as a valid, reliable measure of recovery-related factors, which can complement consumer based instruments in the assessment of changes in recovery.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Spurred by increased interest of members of consumer advocacy groups and recent political influence, professionals in the field of community mental health have undertaken a systemic transformation in order to incorporate the ideas and concepts that facilitate consumers’ recovery (Olmos-Gallo et al. 2012). At the same time, mental health researchers and evaluators have focused on a change from the traditional outcomes (e.g., symptom frequency, recidivism, and hospitalization rates) to recovery-oriented outcomes (Davidson and Roe 2007; Farkas et al. 2005; Olmos-Gallo and DeRoche 2010). Typically, recovery outcomes include the assessment of change in consumers’ perspectives of recovery, commonly defined as changes in: hope, active growth, safety, symptom management, social support, and wellness among others (Lusczakoski et al. 2013; Onken et al. 2007). However, due to the deeply individualized nature of the recovery process (Anthony 2000), direct measurement of this process can be a challenge. In this article, we suggest that in addition to instruments that can be used to measure consumers’ assessment of their own recovery (Lusczakoski et al. 2013; O’Hare et al. 2003; Ridgway 2003; Trauer and Tobias 2004), providers can use measures related to the consumer’s recovery-related factors to assist in the assessment of change in recovery. This paper describes a measure to assess recovery related factors.
As implied by the term, recovery-related factors are actions and events that tend to be correlated with consumers’ recovery, even though the consumers may not necessarily associate them with their own personal journey. In that sense, recovery-related factors can be an indicator or marker of growth in recovery. For example, participation in mental health services may foster recovery, despite the fact that consumers can recover without the involvement/assistance of professional help (Anthony 2000). Similarly, a provider’s assessment of the consumer’s interest in employment or symptom management can be associated with the consumer’s recovery, even though recovery is dependent upon the consumer taking control and responsibility for his/her own life (Jacobson and Curtis 2000). Based upon this rationale, the provider’s assessment is defined as an evaluation of recovery-related factors and not as recovery, per se. In this article, we define recovery-related factors as including: employment, self-education/learning, participation in services, housing, symptom management, active growth, and substance use/abuse. Table 1 presents the operational definitions of these recovery-related factors.
The Recovery Markers Inventory (RMI; developed by the authors) is a short survey completed by case managers/clinicians, intended to measure the recovery-related factors described in Table 1. Initial development of the RMI included a review of the literature and use of data collected from consumer focus groups. This process was supplemented by feedback provided by therapists, consumers, and members of the Mental Health Center of Denver’s (MHCD) Recovery Committee (see: Olmos-Gallo et al. 2012 for background about the MHCD Recovery Committee). RMI items have been used to generate a summed score, implying the measure is unidimensional, but no dimensionality analyses have been conducted to support that use.
The purpose of this study was to determine the unidimensionality, reliability, construct, and convergent validities of the RMI in a sample of adults with severe and persistent mental illness, who received services at a community mental health center at a major metropolitan city in the United States. We estimated reliability and unidimensionality using Rasch analysis (Bond and Fox 2001). We further tested unidimensionality conducting a confirmatory factor analysis in three steps, as suggested by Bollen (1989) and Tabachnick and Fidell (2013): (1) we split the original sample into two random halves, (2) fit a unidimensional model to one of the samples (calibration) and (3) cross-validated results with the second sample (validation). Finally, we tested convergent validity by calculating the correlation between the RMI and two measures of general functioning, and one intended to measure recovery from the consumer’s perspective.
Methods
Participants and Procedure Footnote 1
A total of 1,513 consumers, who had RMIs completed by the case manager/clinician who regularly works with the consumer, in the MHCD Management Information System (MIS) during the month of July 2009, were included in the study. The MIS prompts clinicians/case managers every quarter to complete the instrument for every consumer in his/her caseload, and this reminder will not be removed from the to-do list until the RMI is completed. The RMI scores were matched through use of a unique ID number to demographic information and three other outcomes instruments: the Colorado Client Assessment Record (CCAR; Ellis et al. 1984); the Global Assessment of Functioning (GAF; Greenberg and Rosenheck 2005; Jones et al. 1995); and the Consumer Recovery Measure (CRM; Lusczakoski et al. 2013). These three instruments were completed within 30 days of the RMI. The CCAR is completed by the clinician/case manager who regularly works with the consumer; the GAF is completed by the psychiatrist who oversees the consumer’s treatment during the most recent visit, and the CRM is completed by the consumer him/herself. The participants’ demographic information is presented in Table 2.
Instruments
Recovery Marker Inventory (RMI)
As described earlier, the RMI was designed to measure recovery-related factors defined as the provider’s assessment in the areas of employment, self-education/learning, participation in services, housing, symptom management, active growth, and substance use/abuse. With the exception of substance use/abuse, every area is measured by a single question with a different number of options. The different number of response options is intended to include all the significant measureable changes in the trait of interest. Thus, employment has six options, self-education/learning has 12, participation in services, 6; housing, 11; symptom management, 5; and finally, active growth orientation, has 6 options. It is worth noting that housing has 11 responses because they are also used for State and Federal reporting; however, for scoring purposes, the responses are collapsed into three general categories: (1) unstable/transitional housing (i.e., street, friends/motel homelessness); (2) stable housing (i.e., assisted living, congregate apartments, or single room occupancy); and (3) independent living. The assessment for substance use/abuse requires two steps. First, the provider assesses the consumer’s level of use for ten substances (Alcohol, Cannabis, Cocaine, Hallucinogens, Inhalants, Methamphetamines, Opiates, Over-the-counter, PCP, Sedatives/Hypnotics/Anxiolytics, and Stimulants). Second, if substance use is reported, the consumer’s stage of change is measured for the specific substance (Prochaska et al. 1992). Given that approximately one-half of all consumers report substance use/abuse (National Alliance on Mental Illness 2012), in the present study, the highest level of consumer use among all substances was applied as a moderator. The RMI was completed by the consumer’s regular case manager or clinician, through the on-line Medical Information System.
Colorado Client Assessment Record (CCAR)-Recovery and Global Assessment of Functioning (GAF)
The CCAR is the outcomes instrument used by the Division of Behavioral Health in the State of Colorado to assess mental health (Ellis et al. 1984), and is administered on a scheduled annual basis to all MHCD consumers. A revised version of this instrument includes items which measure: social support, hope, activity involvement, empowerment, and interpersonal relationships. A 9 point rating response scale is used, and this instrument is termed, the CCAR recovery scale (Menefee, personal communication, April 18, 2008). For the current sample, the CCAR recovery scale showed acceptable internal consistency (α = .85).
The GAF scale (Greenberg and Rosenheck 2005; Jones et al. 1995) is used by psychiatrists at MHCD to rate consumers’ functioning on a regular basis (i.e., at least once a year). The GAF is administered as part of the assessment submitted to the Division of Behavioral Health. The scores range from 0 (i.e., lowest functioning) to 100 (i.e., highest functioning). The GAF scores are used to assess overall functioning for individuals who struggle with their mental health, and they are used to assess the concurrent validity of other recovery scales (Fisher et al. 2009). The Office of Behavioral Health at the State level provides training as part of their CCAR package, but no estimates of the reliability of the GAF are yet available. However, strong, statistically significant correlations have been observed against the CCAR’s symptom severity, and overall level of functioning scales (Mahalik personal communication, February 2014). Although no intraclass reliability has been calculated, psychiatrists receive regular refreshers on scoring.
Consumer Recovery Measure (CRM)
The Consumer Recovery Measure (CRM: Lusczakoski et al. 2013) is used to evaluate consumers’ perceptions of their own recovery, and it is completed by consumers every quarter-year at MHCD. The survey consists of 17, 4 point rating response items, from Strongly Agree to Strongly Disagree. It is used to rate consumers’ self-perceptions of hope, safety, symptom management, social network, and active growth. The instrument was developed with use of a Rasch rating scale model (Bond and Fox 2001) and has a person separation reliability of 0.83 and an item separation reliability of 0.96 (Lusczakoski et al. 2013).
Data Analysis
To estimate the psychometric properties of the RMI, three analytical techniques were applied. These were: (a) reliability estimation and construct validity using Rasch analysis; (b) construct validity using confirmatory factor analysis (CFA). We randomly split the original sample into two halves, then fit a unidimensional model to one of the samples (calibration) and cross-validated the results with the second sample (validation); and (c) concurrent validity through Pearson correlations between the RMI and the CCAR Recovery Scale (Menefee 2008, personal communication), the GAF (Greenberg and Rosenheck 2005), and the CRM (Luzczakoski et. al. 2012).
First, a Rasch Partial Credit Model was estimated with the use of Winsteps 3.64 (Linacre, 2007) to determine the person and item reliability and dimensionality, as well as in fit and outfit estimates (Bond and Fox 2001).
Next, the CFA models were estimated with use of LISREL 8.80 (Jöreskog and Sörbom 2006), based on the procedures suggested by Bollen (1989) and Tabachnick and Fidell (2013). CFA is a confirmatory technique used to test a theory about measure structure (Tabachnick and Fidell 2013); in the present case, a single factor defining recovery-related factors. Participants’ were randomly divided into two datasets: (a) a calibration sample (n = 737) to estimate the structure components and (b) a cross-validation sample (n = 776) to verify them (Bollen 1989; Tabachnick and Fidell 2013). To account for the ordinal nature of the items, both CFA models were estimated using polychoric correlation matrices with robust maximum likelihood estimation procedures. Multivariate normality was evaluated and found to hold across all items. Model fit was assessed using Hu and Bentler’s (1999) guidelines: non-normed fit index ≥.95 (NNFI; Bentler and Bonett 1980); comparative fit index ≥.96 (CFI; Bentler 1990); and root mean square error of approximation (RMSEA; Steiger 1989) ≤.05 indicating good fit or ≤.08 for reasonable fit (MacCallum et al. 1996). Finally, Pearson-product moment correlations were computed between the RMI and the CCAR recovery scale, GAF score, and the MHCD CRM with use of SPSS 13 (SPSS Inc 2005).
Results
Rasch Analysis
The RMI showed acceptable reliability estimates, with a person separation reliability of 0.74 and an item separation reliability of 1.00. As displayed in Table 3, all six items in the RMI produced in fit and outfit values within an acceptable range (0.5–1.5). In terms of difficulty to endorse, self-education and employment were the two most difficult items, followed by active growth, participation, symptoms, and housing. As evidence of the unidimensionality, the strength of the first contrast following a principal components analysis of residuals was less than the suggested cutoff value of 2.0 (Bond and Fox 2001; Linacre 2007). In terms of model misfit, the majority of consumers’ Rasch model misfit was due to unexpectedly low scores in the housing item. That is, these consumers improved in every other area, but were still homeless or living in transitional housing.
Construct Validity
The specification of the calibration model included a single latent factor (i.e., recovery-related factor). Regarding substance use, the highest level among all ten substances served as a moderator variable (i.e., exogenous variable). The CFA model structure is presented in Fig. 1. The consumer’s highest level of substance use was negatively correlated to the recovery-related factor (r = −25, p < .01), which suggests that, as a consumer’s substance use increases, the recovery-related factors decrease. Given that the fit statistics were acceptable in the calibration model (NNFI = .96, CFI = .97, RMSEA = .06, CI = .04, .08), the model for the validation sample was specified according to the factor loadings and variance–covariance matrices estimated in the calibration model. The fit of the validation model was also acceptable (NNFI = 1.01, CFI = 1.00, RMSEA < .01, CI < .01, .01). Furthermore, a comparison between the validation and the calibration models showed no significant deviations (S–B χ2 = 0.0 (df = 28), p > .05). The findings from the construct validity analysis supported by Rasch analysis provided evidence that the six items in the RMI can be combined to create a total recovery-related latent factor score, which is moderated by a consumer’s highest level of substance use.
Concurrent Validity Evidence
The RMI exhibited significant correlations with other recovery measures. The strongest relationship was with the CCAR recovery scale (r = −.46, p < .01, 95 % CI −.39, −.52). The RMI showed statistically significant correlations with the GAF (r = .23, p < .01, 95 % CI .15, .32) and the CRM (r = .19, p < .01, 95 % CI .12, .25). It is worth noting that the negative correlation with the CCAR is due to the fact that the CCAR is used to measure severity, not recovery (Fig. 2).
Discussion
The purpose of the study was to investigate the psychometric properties of the RMI, an instrument used to measure the factors associated with recovery from mental illness. As explained in the introduction, the RMI is not intended to measure recovery, but actions and events associated with growth that can be associated with it.
Evidence was provided by this study to support the RMI as a valid and reliable measure of recovery-related factors. The results from this study demonstrated that, with a community-based sample of consumers, the RMI showed (acceptable internal reliability estimates, unidimensionality as a measure of recovery-related factors, and evidence for concurrent validity in the measurement of consumer recovery and general overall functioning. Since the RMI has been used only with consumers receiving services at community-based mental health centers, it is suggested that similar settings can effectively adopt it. It is also worth notice that several of the indicators included under the RMI have been used on a regular basis to report outcomes to the Federal and State governments. Thus, the RMI can play a dual role by: (a) creating an accountability system for external stakeholders and funders; and (b) helping clinical staff, consumers, and their family members to observe improvements over time.
The RMI represents one of several instruments in the multi-pronged measurement approach undertaken by the Mental Health Center of Denver (MHCD) to understand recovery from mental illness. As described elsewhere (Olmos-Gallo et al. 2012), we have to date developed several instruments to explore recovery from multiple perspectives, in a way that can be easily integrated into clinical practice. The combination of multiple instruments has allowed MHCD staff to develop an outcomes-centered approach to recovery, which has already helped to demonstrate significant improvements. For example, since MHCD started to track recovery, with use of the RMI and CRM (Lusczakoski et al. 2013), staff of the two succeed in Employment program has reported a significant increase (38 %) in employment status (Olmos-Gallo et al.), and over 25 % obtained employment within 8 months of enrollment in the program. Approximately 55 % of the consumers in the same program increased their interest in education/learning activities (Outcomes Quarterly, Fall 2010).
The MHCD multi-measure approach has been used to develop a holistic approach toward recovery that takes into account the fact that recovery happens over time, and that there is always the possibility of relapse (Olmos-Gallo and DeRoche 2010). We have developed models with use of hierarchical linear modeling (Raudenbush and Bryk 2002), which can accommodate the hierarchical nature of the longitudinal data collected,Footnote 2 in order to explore the nature of change over time and the potential impact of program characteristics (e.g., modality, intensity) or background variables (e.g., gender, ethnicity, diagnostic). More recently, we have added the use of statistical control charts (Montgomery 2008) that allow multiple stakeholders (e.g., managers, clinicians, consumers) to monitor and provide feedback on consumers’ progress as they move forward in their recovery. Outcomes information is combined with other relevant information (e.g., diagnosis, demographic) in a dashboard that presents it in a user-friendly manner. The combination of valid and reliable recovery-outcome measures, development of predictive models, and statistical control charts has helped MHCD staff to monitor clinical data with an emphasis on quality improvement (McLean et al. 2010).
More importantly, being able to measure recovery from different perspectives allows the staff to ask questions that can help them in their practice. For example: Are medications just relieving symptoms, or helping people improve? Are there some interventions that are better suited to consumers with specific characteristics? What kinds of services help the most when it comes to improving the lives of those with a severe and persistent mental illness? What is the impact of hope in recovery? The answer to these and similar questions will not be possible in the absence of valid and reliable instruments to measure recovery-based outcomes. In this study, we have shown that the RMI holds promise as a reliable and valid measure of factors associated with recovery from mental illness that can be used to answer such questions.
Notes
The project received IRB approval from the University of Northern Colorado committee on the protection of human subjects
For example, in a typical community mental health center, consumers will receive services from the same provider tor clinic.
References
Anthony, W. A. (2000). A recovery-oriented service system: Setting some system standards. Psychiatric Rehabilitation Journal, 24(2), 159–168.
Bentler, P. M. (1990). Comparative fit indexes in structural equation modeling. Quantitative Methods in Psychology, 107, 238–246.
Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88, 588–606.
Bollen, K. A. (1989). Structural equations with latent variables. New York, NY: Wiley.
Bond, T. G., & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in human sciences. Mahwah, NJ: Lawrence Erlbaum Associates.
Davidson, L., & Roe, D. (2007). Recovery from versus recovery in serious mental illness: One strategy for lessening confusion plaguing recovery. Journal of Mental Health, 16, 459–470.
Ellis, R. H., Wilson, N. Z., & Foster, M. F. (1984). Statewide treatment outcomes assessment in Colorado: The Colorado Assessment Record. Community Mental Health Journal, 20, 72–89.
Farkas, M., Gagne, C., Anthony, W., & Chamberlin, J. (2005). Implementing recovery oriented evidence based programs: Identifying the critical dimensions. Community Mental Health Journal, 41(2), 141–157.
Fisher, D., Pilon, D., Hershberger, S., Reynolds, G., LaMaster, S., & Davis, M. (2009). Psychometric properties of an assessment for mental health recovery programs. Community Mental Health Journal, 45(4), 246–250. doi:10.1007/s10597-009-9213-8.
Greenberg, G. A., & Rosenheck, R. A. (2005). Using the GAF as a national mental health outcome measure in the Department of Veterans Affairs. Psychiatric Services, 56, 420–426.
Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structural analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55.
Jacobson, N., & Curtis, L. (2000). Recovery as policy in mental health services: Strategies emerging form the states. Psychiatric Rehabilitation Journal, 23(4), 333–341.
Jones, S. H., Thornicroft, G., Coffey, M., & Dunn, G. (1995). A brief mental health outcome scale-reliability and validity of the Global Assessment of Functioning (GAF). The British Journal of Psychiatry, 166(5), 654–659. doi:10.1192/bjp.166.5.654.
Jöreskog, K. G., & Sörbom, D. (2006). LISREL ® 8: User’s reference guide. Chicago, IL: Scientific Software International.
Linacre, J. M. (2007). Winsteps ® Rasch measurement computer program. Beaverton, OR: Winsteps.com.
Lusczakoski, K. D., Olmos-Gallo, P. A. Milnor, W., & McKinney, C. J. (2013). Measuring mental health recovery: An application of Rasch modeling to the consumer recovery measure. Manuscript submitted for publication.
MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149.
Mahalik, J. (2014). Personal communication, February 13.
McLean, C., Olmos-Gallo, P. A., & McKinney, C. J. (2010, November). The perfect couple, clinical quality and program outcomes: Using data to improve clinical practices. Presented at the 2010 American Evaluation Association Conference. San Antonio, TX.
Menefee, D. (2008). Personal communication, April 18.
Montgomery, D. C. (2008). Statistical quality control: A modern introduction (6th ed.). New York, NY: Wiley.
National Alliance on Mental Illness. (2012). Dual diagnosis and integrated treatment of mental illness and substance abuse disorder. Retrieved from http://www.nami.org/Template.cfm?Section=By_Illness&Template=/TaggedPage/TaggedPageDisplay.cfm&TPLID=54&ContentID=23049.
O’Hare, T., Sherrer, M. V., Connery, H. S., Thornton, J., LaButti, A., & Emrick, K. (2003). Further validation of the Psycho-Social Well-Being Scale (PSWS) with community clients. Community Mental Health Journal, 39(2), 115–129.
Olmos-Gallo, P. A., & DeRoche, K. K. (2010). Monitoring outcomes in mental health recovery: The Effect on programs and policies. Advances in Mental Health, 9(1), 8–16. http://amh.e-contentmanagement.com/archives/vol/9/issue/1/.
Olmos-Gallo, P. A., Starks, R., DeRoche-Lusczakoski, K. K., Huff, S., & Mock, K. (2012). Seven key strategies that work together to create Recovery Based Transformation. Community Mental Health Journal, 48(3), 294–301. doi:10.1007/s10597-011-9441-6.
Onken, S. J., Craig, C. M., Ridgeway, P., Ralph, R. O., & Cook, J. A. (2007). An analysis of the definitions and elements of recovery: A review of the literature. Psychiatric Rehabilitation Journal, 31, 9–22.
Outcomes Quarterly. (2010). http://outcomesmhcd.com/presentations/OutcomesQuarterlyv1i2.pdf.
Prochaska, J. O., DiClemente, C. C., & Norcross, J. C. (1992). In search of how people change: Applications to addictive behaviors. American Psychologist, 47(9), 1102–1114. doi:10.1037/0003-066X.47.9.1102.
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA.: Sage.
Ridgway, P. (2003). The Recovery Enhancing Environment Measure (REE): Using measurement tools to understand and shape recovery-oriented practice. Paper presented at the 2003 Joint National Conference on Mental Health Block Grants and National Conference on Mental Health Statistics, Washington, DC.
SPSS Inc. (2005). SPSS 13.0 for Windows. Chicago, IL: SPSS Inc.
Steiger, J. H. (1989). EZPATH: A supplementary module for SYSTAT and SYGRAPH. Evanston, IL: SYSTAT.
Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Upper Saddle River, N.J.: Pearson.
Trauer, T., & Tobias, G. (2004). The Camberwell assessment of need and behaviour and symptom identification scale as routine outcome measures in a psychiatric disability rehabilitation and support service. Community Mental Health Journal, 40(3), 211–221.
Author information
Authors and Affiliations
Corresponding author
Additional information
Kate DeRoche Lusczakoski and P. Antonio Olmos-Gallo were affiliated with the Mental Health Center of Denver when the research was conducted.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Lusczakoski, K.D., Olmos-Gallo, P.A., McKinney, C.J. et al. Measuring Recovery Related Outcomes: A Psychometric Investigation of the Recovery Markers Inventory. Community Ment Health J 50, 896–902 (2014). https://doi.org/10.1007/s10597-014-9728-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10597-014-9728-5