Skip to main content

Individualization and Quality Improvement: Two New Scales to Complement Measurement of Program Fidelity

We’re sorry, something doesn't seem to be working properly.

Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

Fidelity scales have been widely used to assess program adherence to the principles of an evidence-based practice, but they do not measure important aspects of quality of care. Pragmatic scales measuring clinical quality of services are needed to complement fidelity scales measuring structural aspects of program implementation. As part of the instrumentation developed for the National Implementing Evidence-Based Practices Project, we piloted a new instrument with two 5-item quality scales, Individualization (a client-level quality scale) and Quality Improvement (an organizational-level quality scale). Pairs of independent fidelity assessors conducted fidelity reviews in 49 sites in 8 states at baseline and at four subsequent 6-month intervals over a 2-year follow-up period. The assessors followed a standardized protocol to administer these quality scales during daylong site visits; during these same visits they assessed programs on fidelity to the evidence-based practice that the site was seeking to implement. Assessors achieved acceptable interrater reliability for both Individualization and Quality Improvement. Principal components factor analysis confirmed the 2-scale structure. The two scales were modestly correlated with each other and with the evidence-based practice fidelity scales. Over the first year, Individualization and Quality Improvement improved, but showed little or no improvement during the last year of follow-up. The two newly developed scales showed adequate psychometric properties in this preliminary study, but further research is needed to assess their validity and utility in routine clinical practice.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3

References

  • Adams, N., & Grieder, D. (2004). Treatment planning for person-centered care: The road to mental health and addiction recovery. Amsterdam: Elsevier Academic Press.

    Google Scholar 

  • Arns, P. G., & Linney, J. A. (1995). The relationship of service individualization to client functioning in programs for severely mentally ill persons. Community Mental Health Journal, 31, 127–137. doi:10.1007/BF02188762.

    PubMed  Article  CAS  Google Scholar 

  • Batalden, P. B., & Stoltz, P. K. (1993). A framework for the continual improvement of health care: Building and applying professional and improvement knowledge to test changes in daily work. The Joint Commission Journal on Quality Improvement, 19, 424–452.

    PubMed  CAS  Google Scholar 

  • Becker, D. R., Baker, S. R., Carlson, L., Flint, L., Howell, R., Lindsay, S., et al. (2007). Critical strategies for implementing supported employment. Journal of Vocational Rehabilitation, 27, 13–20.

    Google Scholar 

  • Becker, D. R., Smith, J., Tanzman, B., Drake, R. E., & Tremblay, T. (2001). Fidelity of supported employment programs and employment outcomes. Psychiatric Services (Washington, D.C.), 52, 834–836. doi:10.1176/appi.ps.52.6.834.

    CAS  Google Scholar 

  • Becker, D. R., Xie, H., McHugo, G. J., Halliday, J., & Martinez, R. A. (2006). What predicts supported employment program outcomes? Community Mental Health Journal, 42, 303–313. doi:10.1007/s10597-006-9037-8.

    PubMed  Article  Google Scholar 

  • Bond, G. R., Evans, L., Salyers, M. P., Williams, J., & Kim, H. K. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research, 2, 75–87. doi:10.1023/A:1010153020697.

    PubMed  Article  CAS  Google Scholar 

  • Bond, G. R., Drake, R. E., McHugo, G. J., Rapp, C. A., & Whitley, R. (2009). Strategies for improving fidelity in the National Evidence-Based Practices Project. Research on Social Work Practice. doi:10.1177/1049731509335531.

  • Bond, G. R., McHugo, G. J., Becker, D. R., Rapp, C. A., & Whitley, R. (2008). Fidelity of supported employment: Lessons learned from the National Evidence-Based Practices Project. Psychiatric Rehabilitation Journal, 31, 300–305. doi:10.2975/31.4.2008.300.305.

    PubMed  Article  Google Scholar 

  • Bond, G. R., & Salyers, M. P. (2004). Prediction of outcome from the Dartmouth ACT fidelity scale. CNS Spectrums, 9, 937–942.

    PubMed  Google Scholar 

  • Bond, G. R., Vogler, K. M., Resnick, S. G., Evans, L. J., Drake, R. E., & Becker, D. R. (2001). Dimensions of supported employment: Factor structure of the IPS Fidelity Scale. Journal of Mental Health, 10, 383–393. doi:10.1080/09638230120041146.

    Article  Google Scholar 

  • Brunette, M. F., Asher, D., Whitley, R., Lutz, W. J., Wieder, B. L., Jones, A. M., et al. (2008). Implementation of integrated dual disorders treatment: A qualitative analysis of facilitators and barriers. Psychiatric Services (Washington, D.C.), 59, 989–995. doi:10.1176/appi.ps.59.9.989.

    Google Scholar 

  • Burns, T., Catty, J., Dash, M., Roberts, C., Lockwood, A., & Marshall, M. (2007). Use of intensive case management to reduce time in hospital in people with severe mental illness: Systematic review and meta-regression. British Medical Journal, 335, 336. doi:310.1136/bmj.39251.599259.599255. (18 August, published 13 July 2007).

    PubMed  Article  Google Scholar 

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. doi:10.1037/h0046016.

    PubMed  Article  CAS  Google Scholar 

  • Catty, J., Lissouba, P., White, S., Becker, T., Drake, R. E., Fioritti, A., et al. (2008). Predictors of employment for people with severe mental illness: Results of an international six-centre RCT. The British Journal of Psychiatry, 192, 224–231. doi:10.1192/bjp.bp.107.041475.

    PubMed  Article  Google Scholar 

  • Clarke, S. P., Crowe, T. P., Oades, L. G., & Deane, F. P. (2009). Do goal-setting interventions improve the quality of goals in mental health services? Psychiatric Rehabilitation Journal, 32, 292–299. doi:10.2975/32.4.2009.292.299.

    PubMed  Article  Google Scholar 

  • Cook, J. A. (2004). Blazing trails: Using evidence-based practice and stakeholder consensus to enhance psychosocial rehabilitation services in Texas. Psychiatric Rehabilitation Journal, 27, 305–318. doi:10.2975/27.2004.305.306.

    PubMed  Article  Google Scholar 

  • DeLuca, N. (2007). Consensus on the key ingredients of person-centered treatment planning for adults with severe mental illness. Unpublished dissertation, Indiana University-Purdue University Indianapolis, Indianapolis.

  • Deming, W. E. (1986). Out of crisis. Cambridge, MA: Massachusetts Institute of Technology, Center for Advanced Engineering Study.

    Google Scholar 

  • Drake, R. E., Bond, G. R., & Essock, S. M. (2009). Implementing evidence-based practices for people with schizophrenia. Schizophrenia Bulletin, 35, online.

  • Drake, R. E., Goldman, H. H., Leff, H. S., Lehman, A. F., Dixon, L., Mueser, K. T., et al. (2001). Implementing evidence-based practices in routine mental health service settings. Psychiatric Services (Washington, D.C.), 52, 179–182. doi:10.1176/appi.ps.52.2.179.

    CAS  Google Scholar 

  • Drake, R. E., Rosenberg, S. D., Teague, G. B., Bartels, S. J., & Torrey, W. C. (2003). Fundamental principles of evidence-based medicine applied to mental health care. The Psychiatric Clinics of North America, 24, 811–820. doi:10.1016/S0193-953X(03)00063-7.

    Article  Google Scholar 

  • Druss, B., Rosenheck, R., & Stolar, M. (1999). Patient satisfaction and administrative measures as indicators of the quality of mental health care. Psychiatric Services (Washington, D.C.), 50, 1053–1058.

    CAS  Google Scholar 

  • Finnerty, M. T., Rapp, C. A., Bond, G. R., Lynde, D. W., Ganju, V. J., & Goldman, H. H. (2009). The State Health Authority Yardstick (SHAY). Community Mental Health Journal. doi:10.1007/s10597-009-9181-z.

  • Gowdy, E., & Rapp, C. A. (1989). Managerial behavior: The common denominators of successful community based programs. Psychosocial Rehabilitation Journal, 13(2), 31–51.

    Google Scholar 

  • Hall, L. L., Graf, A. C., Fitzpatrick, M. J., Lane, T., & Birkel, R. C. (2003). Shattered lives: Results of a national survey of NAMI members living with mental illness and their families. Arlington, VA: NAMI/TRIAD (Treatment/Recovery Information and Advocacy Data Base).

    Google Scholar 

  • Harkness, D. R. (1997). Testing interactional social work theory: A panel analysis of supervised practice and outcomes. The Clinical Supervisor, 15, 33–35. doi:10.1300/J001v15n01_03.

    Article  Google Scholar 

  • Harkness, D. R., & Hensley, H. (1991). Changing the focus of social work supervision: Effects on client satisfaction and generalized contentment. Social Work, 36, 506–512.

    PubMed  CAS  Google Scholar 

  • Henggeler, S. W., Pickrel, S. G., & Brondino, M. J. (1999). Multisystemic treatment of substance-abusing and -dependent delinquents: Outcomes, treatment fidelity, and transportability. Mental Health Services Research, 1, 171–184. doi:10.1023/A:1022373813261.

    PubMed  Article  CAS  Google Scholar 

  • Hermann, R. C. (2005). Improving mental healthcare. A guide to measurement-based quality improvement. Washington, DC: American Psychiatric Publishing.

    Google Scholar 

  • Jerrell, J. M., & Ridgely, M. S. (1999). Impact of robustness of program implementation on outcomes of clients in dual diagnosis programs. Psychiatric Services (Washington, D.C.), 50, 109–112.

    CAS  Google Scholar 

  • Lehman, A. F., Kreyenbuhl, J., Buchanan, R. W., Dickerson, F. B., Dixon, L. B., Goldberg, R., et al. (2004). The schizophrenia Patient Outcomes Research Team (PORT): Updated treatment recommendations 2003. Schizophrenia Bulletin, 30, 193–217.

    PubMed  Google Scholar 

  • Lehman, A. F., & Steinwachs, D. M. (1998). Patterns of usual care for schizophrenia: Initial results from the Schizophrenia Patient Outcomes Research Team (PORT) client survey. Schizophrenia Bulletin, 24, 11–23.

    PubMed  CAS  Google Scholar 

  • Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberber, O. (2006). SAS system for mixed models (2nd ed.). Cary, NC: SAS Publishing.

    Google Scholar 

  • Mancini, A. D., Moser, L. L., Whitley, R., McHugo, G. J., Bond, G. R., Finnerty, M. T., et al. (2009). Assertive community treatment: Facilitators and barriers to implementation in routine mental health settings. Psychiatric Services, 60(2), 189–195.

    PubMed  Article  Google Scholar 

  • Marshall, T., Rapp, C. A., Becker, D. R., & Bond, G. R. (2008). Key factors for implementing supported employment. Psychiatric Services (Washington, D.C.), 59, 886–892. doi:10.1176/appi.ps.59.8.886.

    Google Scholar 

  • McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46. doi:10.1037/1082-989X.1.1.30.

    Article  Google Scholar 

  • McGrew, J. H., Johannesen, J. K., Griss, M. E., Born, D. L., & Katuin, C. (2005). Performance-based funding of supported-employment: A multi-site controlled trial. Journal of Vocational Rehabilitation, 23, 81–99.

    Google Scholar 

  • McHugo, G. J., Drake, R. E., Teague, G. B., & Xie, H. (1999). Fidelity to assertive community treatment and client outcomes in the New Hampshire Dual Disorders Study. Psychiatric Services (Washington, D.C.), 50, 818–824.

    CAS  Google Scholar 

  • McHugo, G. J., Drake, R. E., Whitley, R., Bond, G. R., Campbell, K., Rapp, C. A., et al. (2007). Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psychiatric Services (Washington, D.C.), 58, 1279–1284. doi:10.1176/appi.ps.58.10.1279.

    Google Scholar 

  • Mor Barak, M. E., Travis, D. J., Pyun, H., & Xie, B. (2009). The impact of supervision on worker outcomes: A meta-analysis. The Social Service Review, 83, 3–32. doi:10.1086/599028.

    Article  Google Scholar 

  • Nadler, D. A. (1977). Feedback and organization development. Using data based methods. Reading: Addison-Wesley.

    Google Scholar 

  • New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America. Final report. DHHS Pub. No. SMA-03-3832. Rockville, MD: Substance Abuse and Mental Health Services Administration.

    Google Scholar 

  • Ogden, T., Hagen, K. A., Askeland, E., & Christensen, B. (2009). Implementing and evaluating evidence-based treatment of conduct problems in children and youth in Norway. Research on Social Work Practice. doi:10.1177/1049731509335530.

  • Poertner, J., & Rapp, C. A. (2007). Textbook of social administration: The consumer centered approach. Philadelphia, PA: Haworth Press.

    Google Scholar 

  • Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher, D., et al. (2008). Evidence-based practice implementation strategies: Results of a qualitative study. Community Mental Health Journal, 44, 213–224. doi:10.1007/s10597-007-9109-4.

    PubMed  Article  Google Scholar 

  • Rapp, C. A., & Goscha, R. J. (2004). The principles of effective case management of mental health services. Psychiatric Rehabilitation Journal, 27, 319–333. doi:10.2975/27.2004.319.333.

    PubMed  Article  Google Scholar 

  • Salyers, M. P., Bond, G. R., Teague, G. B., Cox, J. F., Smith, M. E., Hick, M. L., et al. (2003). Is it ACT yet? Real-world examples of evaluating the degree of implementation for assertive community treatment. The Journal of Behavioral Health Services & Research, 30, 308–324.

    Google Scholar 

  • SAS. (2008). SAS/STAT user’s guide, version 9.2. Cary, NC: SAS Institute Inc.

    Google Scholar 

  • Shannon, H. S., Robson, L. S., & Sale, J. E. (2001). Creating safer and healthier workplaces: Role of organizational factors and job characteristics. American Journal of Industrial Medicine, 40, 319–334. doi:10.1002/ajim.1106.

    PubMed  Article  CAS  Google Scholar 

  • Simon, G., Rutter, C., Crosier, M., Scott, J., Operskalski, B. H., & Ludman, E. (2009). Are comparisons of consumer satisfaction with providers biased by nonresponse or case-mix differences. Psychiatric Services (Washington, D.C.), 60, 67–73. doi:10.1176/appi.ps.60.1.67.

    Google Scholar 

  • Swift, J. K., & Callahan, J. L. (2009). The impact of client treatment preferences on outcome: A meta-analysis. Journal of Clinical Psychology, 65, 368–381.

    PubMed  Article  Google Scholar 

  • Taylor, M. S. (1987). The effects of feedback on the behavior of organizational personnel. Administration in Social Work, 11, 191–203. doi:10.1300/J147v11n03_18.

    Article  Google Scholar 

  • Torrey, W. C., Lynde, D. W., & Gorman, P. (2005). Promoting the implementation of practices that are supported by research: The National Implementing Evidence-Based Practice Project. Child and Adolescent Psychiatric Clinics of North America, 14, 297–306. doi:10.1016/j.chc.2004.05.004.

    PubMed  Article  Google Scholar 

  • Tsai, J., & Bond, G. R. (2008). A comparison of electronic medical records to conventional paper records in community mental health centers. International Journal for Quality in Health Care, 20, 136–143. doi:10.1093/intqhc/mzm064.

    PubMed  Article  Google Scholar 

  • U.S. Surgeon General. (2000). Mental health: A report of the surgeon general - Executive summary. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health.

    Google Scholar 

  • Wang, P. S., Demler, O., & Kessler, R. C. (2002). Adequacy of treatment for serious mental illness in the United States. American Journal of Public Health, 92, 92–98. doi:10.2105/AJPH.92.1.92.

    PubMed  Article  Google Scholar 

  • Young, A. S., Sullivan, G., Burnam, A., & Brook, R. H. (1998). Measuring the quality of outpatient treatment for schizophrenia. Archives of General Psychiatry, 55, 611–617. doi:10.1001/archpsyc.55.7.611.

    PubMed  Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Gary R. Bond.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Bond, G.R., Drake, R.E., Rapp, C.A. et al. Individualization and Quality Improvement: Two New Scales to Complement Measurement of Program Fidelity. Adm Policy Ment Health 36, 349–357 (2009). https://doi.org/10.1007/s10488-009-0226-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-009-0226-y

Keywords

  • Fidelity
  • Evidence-based practice
  • Quality of care
  • Individualization
  • Quality improvement