Skip to main content

Advertisement

SpringerLink
Log in

Individualization and Quality Improvement: Two New Scales to Complement Measurement of Program Fidelity

  • Original Paper
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Fidelity scales have been widely used to assess program adherence to the principles of an evidence-based practice, but they do not measure important aspects of quality of care. Pragmatic scales measuring clinical quality of services are needed to complement fidelity scales measuring structural aspects of program implementation. As part of the instrumentation developed for the National Implementing Evidence-Based Practices Project, we piloted a new instrument with two 5-item quality scales, Individualization (a client-level quality scale) and Quality Improvement (an organizational-level quality scale). Pairs of independent fidelity assessors conducted fidelity reviews in 49 sites in 8 states at baseline and at four subsequent 6-month intervals over a 2-year follow-up period. The assessors followed a standardized protocol to administer these quality scales during daylong site visits; during these same visits they assessed programs on fidelity to the evidence-based practice that the site was seeking to implement. Assessors achieved acceptable interrater reliability for both Individualization and Quality Improvement. Principal components factor analysis confirmed the 2-scale structure. The two scales were modestly correlated with each other and with the evidence-based practice fidelity scales. Over the first year, Individualization and Quality Improvement improved, but showed little or no improvement during the last year of follow-up. The two newly developed scales showed adequate psychometric properties in this preliminary study, but further research is needed to assess their validity and utility in routine clinical practice.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • Adams, N., & Grieder, D. (2004). Treatment planning for person-centered care: The road to mental health and addiction recovery. Amsterdam: Elsevier Academic Press.

    Google Scholar 

  • Arns, P. G., & Linney, J. A. (1995). The relationship of service individualization to client functioning in programs for severely mentally ill persons. Community Mental Health Journal, 31, 127–137. doi:10.1007/BF02188762.

    Article  PubMed  CAS  Google Scholar 

  • Batalden, P. B., & Stoltz, P. K. (1993). A framework for the continual improvement of health care: Building and applying professional and improvement knowledge to test changes in daily work. The Joint Commission Journal on Quality Improvement, 19, 424–452.

    PubMed  CAS  Google Scholar 

  • Becker, D. R., Baker, S. R., Carlson, L., Flint, L., Howell, R., Lindsay, S., et al. (2007). Critical strategies for implementing supported employment. Journal of Vocational Rehabilitation, 27, 13–20.

    Google Scholar 

  • Becker, D. R., Smith, J., Tanzman, B., Drake, R. E., & Tremblay, T. (2001). Fidelity of supported employment programs and employment outcomes. Psychiatric Services (Washington, D.C.), 52, 834–836. doi:10.1176/appi.ps.52.6.834.

    CAS  Google Scholar 

  • Becker, D. R., Xie, H., McHugo, G. J., Halliday, J., & Martinez, R. A. (2006). What predicts supported employment program outcomes? Community Mental Health Journal, 42, 303–313. doi:10.1007/s10597-006-9037-8.

    Article  PubMed  Google Scholar 

  • Bond, G. R., Evans, L., Salyers, M. P., Williams, J., & Kim, H. K. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research, 2, 75–87. doi:10.1023/A:1010153020697.

    Article  PubMed  CAS  Google Scholar 

  • Bond, G. R., Drake, R. E., McHugo, G. J., Rapp, C. A., & Whitley, R. (2009). Strategies for improving fidelity in the National Evidence-Based Practices Project. Research on Social Work Practice. doi:10.1177/1049731509335531.

  • Bond, G. R., McHugo, G. J., Becker, D. R., Rapp, C. A., & Whitley, R. (2008). Fidelity of supported employment: Lessons learned from the National Evidence-Based Practices Project. Psychiatric Rehabilitation Journal, 31, 300–305. doi:10.2975/31.4.2008.300.305.

    Article  PubMed  Google Scholar 

  • Bond, G. R., & Salyers, M. P. (2004). Prediction of outcome from the Dartmouth ACT fidelity scale. CNS Spectrums, 9, 937–942.

    PubMed  Google Scholar 

  • Bond, G. R., Vogler, K. M., Resnick, S. G., Evans, L. J., Drake, R. E., & Becker, D. R. (2001). Dimensions of supported employment: Factor structure of the IPS Fidelity Scale. Journal of Mental Health, 10, 383–393. doi:10.1080/09638230120041146.

    Article  Google Scholar 

  • Brunette, M. F., Asher, D., Whitley, R., Lutz, W. J., Wieder, B. L., Jones, A. M., et al. (2008). Implementation of integrated dual disorders treatment: A qualitative analysis of facilitators and barriers. Psychiatric Services (Washington, D.C.), 59, 989–995. doi:10.1176/appi.ps.59.9.989.

    Google Scholar 

  • Burns, T., Catty, J., Dash, M., Roberts, C., Lockwood, A., & Marshall, M. (2007). Use of intensive case management to reduce time in hospital in people with severe mental illness: Systematic review and meta-regression. British Medical Journal, 335, 336. doi:310.1136/bmj.39251.599259.599255. (18 August, published 13 July 2007).

    Article  PubMed  Google Scholar 

  • Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. doi:10.1037/h0046016.

    Article  PubMed  CAS  Google Scholar 

  • Catty, J., Lissouba, P., White, S., Becker, T., Drake, R. E., Fioritti, A., et al. (2008). Predictors of employment for people with severe mental illness: Results of an international six-centre RCT. The British Journal of Psychiatry, 192, 224–231. doi:10.1192/bjp.bp.107.041475.

    Article  PubMed  Google Scholar 

  • Clarke, S. P., Crowe, T. P., Oades, L. G., & Deane, F. P. (2009). Do goal-setting interventions improve the quality of goals in mental health services? Psychiatric Rehabilitation Journal, 32, 292–299. doi:10.2975/32.4.2009.292.299.

    Article  PubMed  Google Scholar 

  • Cook, J. A. (2004). Blazing trails: Using evidence-based practice and stakeholder consensus to enhance psychosocial rehabilitation services in Texas. Psychiatric Rehabilitation Journal, 27, 305–318. doi:10.2975/27.2004.305.306.

    Article  PubMed  Google Scholar 

  • DeLuca, N. (2007). Consensus on the key ingredients of person-centered treatment planning for adults with severe mental illness. Unpublished dissertation, Indiana University-Purdue University Indianapolis, Indianapolis.

  • Deming, W. E. (1986). Out of crisis. Cambridge, MA: Massachusetts Institute of Technology, Center for Advanced Engineering Study.

    Google Scholar 

  • Drake, R. E., Bond, G. R., & Essock, S. M. (2009). Implementing evidence-based practices for people with schizophrenia. Schizophrenia Bulletin, 35, online.

  • Drake, R. E., Goldman, H. H., Leff, H. S., Lehman, A. F., Dixon, L., Mueser, K. T., et al. (2001). Implementing evidence-based practices in routine mental health service settings. Psychiatric Services (Washington, D.C.), 52, 179–182. doi:10.1176/appi.ps.52.2.179.

    CAS  Google Scholar 

  • Drake, R. E., Rosenberg, S. D., Teague, G. B., Bartels, S. J., & Torrey, W. C. (2003). Fundamental principles of evidence-based medicine applied to mental health care. The Psychiatric Clinics of North America, 24, 811–820. doi:10.1016/S0193-953X(03)00063-7.

    Article  Google Scholar 

  • Druss, B., Rosenheck, R., & Stolar, M. (1999). Patient satisfaction and administrative measures as indicators of the quality of mental health care. Psychiatric Services (Washington, D.C.), 50, 1053–1058.

    CAS  Google Scholar 

  • Finnerty, M. T., Rapp, C. A., Bond, G. R., Lynde, D. W., Ganju, V. J., & Goldman, H. H. (2009). The State Health Authority Yardstick (SHAY). Community Mental Health Journal. doi:10.1007/s10597-009-9181-z.

  • Gowdy, E., & Rapp, C. A. (1989). Managerial behavior: The common denominators of successful community based programs. Psychosocial Rehabilitation Journal, 13(2), 31–51.

    Google Scholar 

  • Hall, L. L., Graf, A. C., Fitzpatrick, M. J., Lane, T., & Birkel, R. C. (2003). Shattered lives: Results of a national survey of NAMI members living with mental illness and their families. Arlington, VA: NAMI/TRIAD (Treatment/Recovery Information and Advocacy Data Base).

    Google Scholar 

  • Harkness, D. R. (1997). Testing interactional social work theory: A panel analysis of supervised practice and outcomes. The Clinical Supervisor, 15, 33–35. doi:10.1300/J001v15n01_03.

    Article  Google Scholar 

  • Harkness, D. R., & Hensley, H. (1991). Changing the focus of social work supervision: Effects on client satisfaction and generalized contentment. Social Work, 36, 506–512.

    PubMed  CAS  Google Scholar 

  • Henggeler, S. W., Pickrel, S. G., & Brondino, M. J. (1999). Multisystemic treatment of substance-abusing and -dependent delinquents: Outcomes, treatment fidelity, and transportability. Mental Health Services Research, 1, 171–184. doi:10.1023/A:1022373813261.

    Article  PubMed  CAS  Google Scholar 

  • Hermann, R. C. (2005). Improving mental healthcare. A guide to measurement-based quality improvement. Washington, DC: American Psychiatric Publishing.

    Google Scholar 

  • Jerrell, J. M., & Ridgely, M. S. (1999). Impact of robustness of program implementation on outcomes of clients in dual diagnosis programs. Psychiatric Services (Washington, D.C.), 50, 109–112.

    CAS  Google Scholar 

  • Lehman, A. F., Kreyenbuhl, J., Buchanan, R. W., Dickerson, F. B., Dixon, L. B., Goldberg, R., et al. (2004). The schizophrenia Patient Outcomes Research Team (PORT): Updated treatment recommendations 2003. Schizophrenia Bulletin, 30, 193–217.

    PubMed  Google Scholar 

  • Lehman, A. F., & Steinwachs, D. M. (1998). Patterns of usual care for schizophrenia: Initial results from the Schizophrenia Patient Outcomes Research Team (PORT) client survey. Schizophrenia Bulletin, 24, 11–23.

    PubMed  CAS  Google Scholar 

  • Littell, R. C., Milliken, G. A., Stroup, W. W., Wolfinger, R. D., & Schabenberber, O. (2006). SAS system for mixed models (2nd ed.). Cary, NC: SAS Publishing.

    Google Scholar 

  • Mancini, A. D., Moser, L. L., Whitley, R., McHugo, G. J., Bond, G. R., Finnerty, M. T., et al. (2009). Assertive community treatment: Facilitators and barriers to implementation in routine mental health settings. Psychiatric Services, 60(2), 189–195.

    Article  PubMed  Google Scholar 

  • Marshall, T., Rapp, C. A., Becker, D. R., & Bond, G. R. (2008). Key factors for implementing supported employment. Psychiatric Services (Washington, D.C.), 59, 886–892. doi:10.1176/appi.ps.59.8.886.

    Google Scholar 

  • McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46. doi:10.1037/1082-989X.1.1.30.

    Article  Google Scholar 

  • McGrew, J. H., Johannesen, J. K., Griss, M. E., Born, D. L., & Katuin, C. (2005). Performance-based funding of supported-employment: A multi-site controlled trial. Journal of Vocational Rehabilitation, 23, 81–99.

    Google Scholar 

  • McHugo, G. J., Drake, R. E., Teague, G. B., & Xie, H. (1999). Fidelity to assertive community treatment and client outcomes in the New Hampshire Dual Disorders Study. Psychiatric Services (Washington, D.C.), 50, 818–824.

    CAS  Google Scholar 

  • McHugo, G. J., Drake, R. E., Whitley, R., Bond, G. R., Campbell, K., Rapp, C. A., et al. (2007). Fidelity outcomes in the National Implementing Evidence-Based Practices Project. Psychiatric Services (Washington, D.C.), 58, 1279–1284. doi:10.1176/appi.ps.58.10.1279.

    Google Scholar 

  • Mor Barak, M. E., Travis, D. J., Pyun, H., & Xie, B. (2009). The impact of supervision on worker outcomes: A meta-analysis. The Social Service Review, 83, 3–32. doi:10.1086/599028.

    Article  Google Scholar 

  • Nadler, D. A. (1977). Feedback and organization development. Using data based methods. Reading: Addison-Wesley.

    Google Scholar 

  • New Freedom Commission on Mental Health. (2003). Achieving the promise: Transforming mental health care in America. Final report. DHHS Pub. No. SMA-03-3832. Rockville, MD: Substance Abuse and Mental Health Services Administration.

    Google Scholar 

  • Ogden, T., Hagen, K. A., Askeland, E., & Christensen, B. (2009). Implementing and evaluating evidence-based treatment of conduct problems in children and youth in Norway. Research on Social Work Practice. doi:10.1177/1049731509335530.

  • Poertner, J., & Rapp, C. A. (2007). Textbook of social administration: The consumer centered approach. Philadelphia, PA: Haworth Press.

    Google Scholar 

  • Rapp, C. A., Etzel-Wise, D., Marty, D., Coffman, M., Carlson, L., Asher, D., et al. (2008). Evidence-based practice implementation strategies: Results of a qualitative study. Community Mental Health Journal, 44, 213–224. doi:10.1007/s10597-007-9109-4.

    Article  PubMed  Google Scholar 

  • Rapp, C. A., & Goscha, R. J. (2004). The principles of effective case management of mental health services. Psychiatric Rehabilitation Journal, 27, 319–333. doi:10.2975/27.2004.319.333.

    Article  PubMed  Google Scholar 

  • Salyers, M. P., Bond, G. R., Teague, G. B., Cox, J. F., Smith, M. E., Hick, M. L., et al. (2003). Is it ACT yet? Real-world examples of evaluating the degree of implementation for assertive community treatment. The Journal of Behavioral Health Services & Research, 30, 308–324.

    Google Scholar 

  • SAS. (2008). SAS/STAT user’s guide, version 9.2. Cary, NC: SAS Institute Inc.

    Google Scholar 

  • Shannon, H. S., Robson, L. S., & Sale, J. E. (2001). Creating safer and healthier workplaces: Role of organizational factors and job characteristics. American Journal of Industrial Medicine, 40, 319–334. doi:10.1002/ajim.1106.

    Article  PubMed  CAS  Google Scholar 

  • Simon, G., Rutter, C., Crosier, M., Scott, J., Operskalski, B. H., & Ludman, E. (2009). Are comparisons of consumer satisfaction with providers biased by nonresponse or case-mix differences. Psychiatric Services (Washington, D.C.), 60, 67–73. doi:10.1176/appi.ps.60.1.67.

    Google Scholar 

  • Swift, J. K., & Callahan, J. L. (2009). The impact of client treatment preferences on outcome: A meta-analysis. Journal of Clinical Psychology, 65, 368–381.

    Article  PubMed  Google Scholar 

  • Taylor, M. S. (1987). The effects of feedback on the behavior of organizational personnel. Administration in Social Work, 11, 191–203. doi:10.1300/J147v11n03_18.

    Article  Google Scholar 

  • Torrey, W. C., Lynde, D. W., & Gorman, P. (2005). Promoting the implementation of practices that are supported by research: The National Implementing Evidence-Based Practice Project. Child and Adolescent Psychiatric Clinics of North America, 14, 297–306. doi:10.1016/j.chc.2004.05.004.

    Article  PubMed  Google Scholar 

  • Tsai, J., & Bond, G. R. (2008). A comparison of electronic medical records to conventional paper records in community mental health centers. International Journal for Quality in Health Care, 20, 136–143. doi:10.1093/intqhc/mzm064.

    Article  PubMed  Google Scholar 

  • U.S. Surgeon General. (2000). Mental health: A report of the surgeon general - Executive summary. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health.

    Google Scholar 

  • Wang, P. S., Demler, O., & Kessler, R. C. (2002). Adequacy of treatment for serious mental illness in the United States. American Journal of Public Health, 92, 92–98. doi:10.2105/AJPH.92.1.92.

    Article  PubMed  Google Scholar 

  • Young, A. S., Sullivan, G., Burnam, A., & Brook, R. H. (1998). Measuring the quality of outpatient treatment for schizophrenia. Archives of General Psychiatry, 55, 611–617. doi:10.1001/archpsyc.55.7.611.

    Article  PubMed  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Psychology, Indiana University Purdue University Indianapolis, Indianapolis, IN, USA

    Gary R. Bond

  2. Department of Psychiatry, Dartmouth Medical School and Dartmouth Psychiatric Research Center, Lebanon, NH, USA

    Robert E. Drake

  3. School of Social Welfare, University of Kansas, Lawrence, KS, USA

    Charles A. Rapp

  4. Department of Community and Family Medicine, Dartmouth Medical School & Dartmouth Psychiatric Research Center, Lebanon, NH, USA

    Robert E. Drake, Gregory J. McHugo & Haiyi Xie

Authors
  1. Gary R. Bond
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robert E. Drake
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charles A. Rapp
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gregory J. McHugo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Haiyi Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gary R. Bond.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bond, G.R., Drake, R.E., Rapp, C.A. et al. Individualization and Quality Improvement: Two New Scales to Complement Measurement of Program Fidelity. Adm Policy Ment Health 36, 349–357 (2009). https://doi.org/10.1007/s10488-009-0226-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-009-0226-y

Keywords

Search

Navigation