Skip to main content
Log in

When Programs Benefit Some People More than Others: Tests of Differential Service Effectiveness

Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Practitioners need to know for whom evidence-based services are most or least effective, but few services research studies provide this information. Using data from a randomized controlled comparison of supported employment findings for two multi-service psychiatric rehabilitation programs, we illustrate and compare procedures for measuring program-by-client characteristic interactions depicting differential program effectiveness, and then illustrate how a significant program-by-client interaction can explain overall program differences in service effectiveness. Interaction analyses based on cluster analysis-identified sample subgroups appear to provide statistically powerful and meaningful hypothesis tests that can aid in the interpretation of main effect findings and help to refine program theory.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abel, K. M., Webb, R. T., Salmon, M. P., Wan, M. W., & Appleby, L. (2005). Prevalence and predictors of parenting outcomes in a cohort of mothers with schizophrenia admitted for joint mother and baby psychiatric care in England. Journal of Clinical Psychiatry, 66(6), 781–789.

    Article  PubMed  Google Scholar 

  • Ackerman, R. J. (1999). An interactional approach for pharmacopsychologists and psychologists: Gender concerns. Journal of Clinical Psychology in Medical Settings, 6(1), 39–61.

    Article  Google Scholar 

  • Aguinis, H. (2004). Regression analysis for categorical moderators. New York: Guilford Press.

    Google Scholar 

  • Aguinis, H., & Stone-Romero, E. F. (1997). Methodological artifacts in moderated multiple regression and their effects on statistical power. Journal of Applied Psychology, 82(1), 192–201.

    Article  Google Scholar 

  • Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.

    Google Scholar 

  • Allness, D., & Knoedler, W. (1998). The PACT model: A Manual for PACT Start-Up. Arlington, VA: National Alliance for the Mentally Ill.

    Google Scholar 

  • Anderson, S. B. (1998). We are not alone: Fountain house and the development of clubhouse culture. New York, NY: Fountain House, Inc.

    Google Scholar 

  • Anthony, W. A., Rogers, E. S., Cohen, M., & Davies, R. R. (1995). Relationships between psychiatric symptomatology, work skills, and future vocational performance. Psychiatric Services, 46(4), 353–358.

    PubMed  CAS  Google Scholar 

  • Aronson, E., Ellsworth, P., Carlsmith, J. M., & Gonzales, M. H. (1990). Methods of research in social psychology. New York: McGraw-Hill.

    Google Scholar 

  • Batstra, L., Bos, E. H., & Neeleman, J. (2002). Quantifying psychiatric comorbidity: Lessons from chronic disease epidemiology. Social Psychiatry and Psychiatric Epidemiology, 37, 105–111.

    Article  PubMed  CAS  Google Scholar 

  • Beard, J. H., Propst, R. N., & Malamud, T. J. (1982). The fountain house model of psychiatric rehabilitation. Psychosocial Rehabilitation Journal, 5(1), 1–12.

    Google Scholar 

  • Bekker, M. H. J. (2003). Investigating gender within health research is more than sex disaggregation of data: A multi-facet gender and health model. Psychology, Health and Medicine, 8(2), 231–243.

    Article  Google Scholar 

  • Beutler, L., Brown, M., Crothers, L., Booker, K., & Seabrook, M. (1996). The dilemma of factitious demographic distinctions in psychological research. Journal of Consulting and Clinical Psychology, 64, 892–902.

    Article  PubMed  CAS  Google Scholar 

  • Bickman, L., Noser, K., & Summerfelt, W. T. (1999). Long-term effects of a system of care on children and adolescents. Journal of Behavioral Health Services and Research, 26(2), 185–202.

    Article  PubMed  CAS  Google Scholar 

  • Blankertz, L. (1998). The value and practicality of deliberate sampling for heterogeneity: A critical multiplist perspective. American JoumaI of Evaluation, 19(3), 307–324.

    Google Scholar 

  • Bond, G. R., Vogler, K., Resnick, S. G., Evans, L., Drake, R., & Becker, D. (2001). Dimensions of supported employment: Factor structure of the IPS fidelity scale. Journal of Mental Health (UK), 10(4), 383–393.

    Article  Google Scholar 

  • Bühringer, G. (2006). Allocating treatment options to patient profiles: Clinical art or science? Addiction, 101, 646–652.

    Article  PubMed  Google Scholar 

  • Burke-Miller, J., Cook, J. A., Grey, D. D., Razzano, L., Blyler, C. R., Leff, H. S., et al. (2006). Demographic characteristics and employment among people with severe mental illness in a multisite study. Community Mental Health Journal, 42(2), 143–159.

    Article  PubMed  Google Scholar 

  • Carey, K. B., Henson, J. M., Carey, M. P., & Maisto, S. A. (2007). Which heavy drinking college students benefit from a brief motivational intervention? Journal of Consulting and Clinical Psychology, 75(4), 663–669.

    Article  PubMed  Google Scholar 

  • Chambless, D. L., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66(1), 7–18.

    Article  PubMed  CAS  Google Scholar 

  • Chwastiak, L. A., Rosenheck, R. A., McEvoy, J. P., Keefe, R. S. E., Swartz, M. S., & Lieberman, J. A. (2006). Interrelationships of psychiatric symptom severity, medical comorbidity, and functioning in schizophrenia. Psychiatric Services, 57(8), 1102–1109.

    Article  PubMed  Google Scholar 

  • Clark, C., & Rich, A. R. (2003). Outcomes of homeless adults with mental illness in a housing program and in case management only. Psychiatric Services, 54(1), 78–83.

    Article  PubMed  Google Scholar 

  • Cook, J. A., Pickett-Schenk, S. A., Gray, D., Banghart, M., Rosenheck, R., & Randolph, F. (2001). Vocational outcomes among formerly homeless persons with severe mental illness in the ACCESS program. Psychiatric Services, 52(8), 1075–1080.

    Article  PubMed  CAS  Google Scholar 

  • Cook, J. A., Leff, H. S., Blyler, C. R., Gold, P. B., Goldberg, R. W., Mueser, K. T., et al. (2005). Results of a multisite randomized trial of supported employment interventions for individuals with severe mental illness. Archives of General Psychiatry, 62, 505–512.

    Article  PubMed  Google Scholar 

  • Dembling, B. P., Chen, D. T., & Vachon, L. (1999). Life expectancy and causes of death in a population treated for serious mental illness. Psychiatric Services, 50(8), 1036–1042.

    PubMed  CAS  Google Scholar 

  • Department of Labor. (1998). Job Training Program Act, Disability Grant Program Funded Under Title III, Section 323, and Title IV, Part D, Section 452 (Vol. 63): Federal Register.

  • Dickey, B., Normand, S.-L., Weiss, R., Drake, R., & Azeni, H. (2002). Medical morbidity, mental illness, and substance use disorders. Psychiatric Services, 53(7), 861–867.

    Article  PubMed  Google Scholar 

  • Dixon, L., Goldberg, R., Lehman, A., & McNary, S. (2001). The impact of health status on work, symptoms, and functional outcomes in severe mental illness. Journal of Nervous and Mental Disease, 189, 17–23.

    Article  PubMed  CAS  Google Scholar 

  • Drebing, C., Van Ormer, A., Krebs, C., Rosenheck, R., Rounsaville, B., Herz, L., et al. (2005). The impact of enhanced incentives on vocational rehabilitation outcomes for dually diagnosed veterans. Journal of Applied Behavior Analysis, 38(3), 359–372.

    Article  PubMed  Google Scholar 

  • Druss, B. G., Marcus, S. C., Rosenheck, R. A., Olfson, M., Tanielian, T., & Pincus, H. A. (2000). Understanding disability in mental and general medical conditions. American Journal of Psychiatry, 157(9), 1485–1491.

    Article  PubMed  CAS  Google Scholar 

  • Frey, J. L. (1994). Long term support: The critical element to sustaining competitive employment: Where do we begin? Psychosocial Rehabilitation Journal, 17(3), 127–133.

    Google Scholar 

  • Goldberg, R. W., Lucksted, A., McNary, S., Gold, J. M., Dixon, L., & Lehman, A. (2001). Correlates of long-term unemployment among inner-city adults with serious and persistent mental illness. Psychiatric Services, 52(1), 101–103.

    Article  PubMed  CAS  Google Scholar 

  • Halvorsen, M. S., & Monsen, J. T. (2007). Self-image as a moderator of change in psychotherapy. Psychotherapy Research, 17(2), 205–217.

    Article  Google Scholar 

  • Hendryx, M. S., Moore, R., Leeper, T., Reynolds, M., & Davis, S. (2001). An examination of methods for risk-adjustment of rehospitalization rates. Mental Health Services Research, 3(1), 15–24.

    Article  PubMed  CAS  Google Scholar 

  • Hendryx, M. S., & Teague, G. B. (2001). Comparing alternative risk-adjustment models. Journal of Behavioral Health Services and Research, 28(3), 247–257.

    Article  PubMed  CAS  Google Scholar 

  • Hodges, K., Xue, Y., & Wotring, J. (2004). Outcomes for children with problematic behavior in school and at home served by public mental health. Journal of Emotional and Behavioral Disorders, 12(2), 109–119.

    Article  Google Scholar 

  • Howell, W., & Peterson, P. (2004). Uses of theory in randomized field trials. American Behavioral Scientist, 47(5), 634–657.

    Article  Google Scholar 

  • James, G. M., Sugar, C. A., Desai, R., & Rosenheck, R. A. (2006). A comparison of outcomes among patients with schizophrenia in two mental health systems: A health state approach. Schizophrenia Research, 86, 309–320.

    Article  PubMed  Google Scholar 

  • Jensen, P., Arnold, E., Richters, J., Severe, J., Vereen, D., Vittiello, B., & Shiller, E. (1999). Moderators and mediators of treatment response for children with attention-deficit/hyperactivity disorder. Archives of General Psychiatry, 56, 1088–1096.

    Article  Google Scholar 

  • Jones, D. R., Macias, C., Barreira, P. J., Fisher, W. H., Hargreaves, W. A., & Harding, C. M. (2004). Prevalence, severity, and co-occurrence of chronic physical health problems of persons with serious mental illness. Psychiatric Services, 55(11), 1250–1257.

    Article  PubMed  Google Scholar 

  • Judd, C. M., & Kenny, D. A. (1981). Estimating the effects of social interventions. New York: Cambridge University Press.

    Google Scholar 

  • Kay, S. R., Fiszbein, A., & Opler, L. A. (1987). The positive and negative syndrome scale (PANSS) for schizophrenia. Schizophrenia Bulletin, 13, 261–276.

    PubMed  CAS  Google Scholar 

  • Kenny, D. A., Calsyn, R. J., Morse, G. A., Klinkenberg, W. D., Winter, J. P., & Trusty, M. L. (2004). Evaluation of treatment programs for persons with severe mental illness: Moderator and mediator effects. Evaluation Review, 28(4), 294–324.

    Article  PubMed  Google Scholar 

  • King, R. D., Gaines, L. S., Lambert, E. W., Summerfelt, W. T., & Bickman, L. (2000). The co-occurrence of psychiatric substance use diagnoses in adolescents in different service systems: Frequency, recognition, cost, and outcomes. Journal of Behavioral Health Services and Research, 27(4), 417–430.

    Article  PubMed  CAS  Google Scholar 

  • Kraemer, H. C., Stice, E., Kazdin, A., Offord, D., & Kupfer, D. (2001). How do risk factors work together? Mediators, moderators, and independent, overlapping, and proxy risk factors. American Journal of Psychiatry, 158(6), 848–856.

    Article  PubMed  CAS  Google Scholar 

  • Kraemer, H. C., Wilson, G. T., Fairburn, C. G., & Agras, W. S. (2002). Mediators and moderators of treatment effects in randomized clinical trials. Archives of General Psychiatry, 59, 877–883.

    Article  PubMed  Google Scholar 

  • Kraemer, H. C., Wilson, K. A., & Hayward, C. (2006). Lifetime prevalence and pseudocomorbidity in psychiatric research. Archives of General Psychiatry, 63, 604–608.

    Article  PubMed  Google Scholar 

  • Krause, M. S., & Howard, K. I. (2003). What random assignment does and does not do. Journal of Clinical Psychology, 59, 751–766.

    Article  PubMed  Google Scholar 

  • Kronick, R., Gilmer, T., Dreyfus, T., & Lee, L. (2000). Improving health-based payment for medicaid beneficiaries: CDPS. Health Care Financing Review, 21(3), 29–64.

    PubMed  CAS  Google Scholar 

  • Lehman, A. F., Goldberg, R. W., Dixon, L. B., NcNary, S., Postrado, L., Hackman, A., et al. (2002). Improving employment outcomes for persons with severe mental illnesses. Archives of General Psychiatry, 59(2), 165–172.

    Article  PubMed  Google Scholar 

  • Lipchik, G. L., Nicholson, R. A., & Penzien, D. B. (2005). Allocation of patients to conditions in headache clinical trials: Randomization, stratification, and treatment matching. Headache, 45, 419–428.

    Article  PubMed  Google Scholar 

  • Macias, C., DeCarlo, L., Wang, Q., Frey, J., & Barreira, P. (2001). Work interest as a predictor of competitive employment: Policy implications for psychiatric rehabilitation. Administration and Policy in Mental Health, 28(4), 279–297.

    Article  PubMed  CAS  Google Scholar 

  • Macias, C., Barreira, P., Hargreaves, W., Bickman, L., Fisher, W. H., & Aronson, E. (2005). Impact of referral source and study applicants’ preference for randomly assigned service on research enrollment, service engagement, and evaluative outcomes. American Journal of Psychiatry, 162(4), 781–787.

    Article  PubMed  Google Scholar 

  • Macias, C., Rodican, C. F., Hargreaves, W. A., Jones, D. R., Barreira, P. J., & Wang, Q. (2006). Supported employment outcomes of a randomized controlled trial of assertive community treatment and clubhouse models. Psychiatric Services, 57(10), 1406–1415.

    Article  PubMed  Google Scholar 

  • Maisto, S. A., Conigliaro, J., McNeil, M., Kraemer, K., & Kelley, M. E. (2001). The relationship between eligibility criteria for participation in alcohol brief intervention trials and other alcohol and health-related variables. American Journal of Addictions, 10(3), 218–231.

    Article  CAS  Google Scholar 

  • McBride, C., Atkinson, L., Quilty, L. C., & Bagby, R. M. (2006). Attachment as moderator of treatment outcome in major depression: A randomized control trial of interpersonal psychotherapy versus cognitive behavior therapy. Journal of Consulting and Clinical Psychology, 74(6), 1041–1054.

    Article  PubMed  Google Scholar 

  • McKendrick, K., Sullivan, C., Banks, S., & Sacks, S. (2007). Modified therapeutic community treatment for offenders with MICA disorders: Antisocial personality disorder and treatment outcomes. Journal of Offender Rehabilitation, 44(2–3), 133–159.

    Article  Google Scholar 

  • Miklowitz, D., & Clarkin, J. F. (1999). Balancing internal and external validity. Prevention and Treatment, 2, 1–4.

    Google Scholar 

  • Mueser, K. T., Salyers, M. P., & Mueser, P. R. (2001). A prospective analysis of work in schizophrenia. Schizophrenia Bulletin, 27(2), 281–296.

    PubMed  CAS  Google Scholar 

  • Ogrodniczuk, J. S., Joyce, A. S., & Piper, W. E. (2007). Effect of patient dissatisfaction with the therapist on group therapy outcome. Clinical Psychology and Psychotherapy, 14(2), 126–134.

    Article  Google Scholar 

  • Peck, L. R. (2005). Using cluster analysis in program evaluation. Evaluation Review, 29(2), 178–196.

    Article  PubMed  Google Scholar 

  • Pettinati, H. M., Volpicelli, J. R., Kranzler, H. R., Luck, G., Rukstalis, M. R., & Cnaan, A. (2000). Sertraline treatment for alcohol dependence: Interactive effects of medication and alcoholic subtype. Alcoholism: Clinical and Experimental Research, 24(7), 1041–1049.

    Article  CAS  Google Scholar 

  • Propst, R. (1992). Standards for clubhouse programs: Why and how they were developed. Psychosocial Rehabilitation Journal, 16(2), 25–30.

    Google Scholar 

  • Rapkin, B. D., & Dumont, K. A. (2000). Methods for identifying and assessing groups in health behavioral research. Addiction, 95(Supplement 3), S395–S417.

    Article  PubMed  Google Scholar 

  • Razzano, L., & Hamilton, M. M. (2005). Health-related barriers to employment among people with HIV/AIDS. Journal of Vocational Rehabilitation, 22(3), 179–188.

    Google Scholar 

  • Razzano, L. A., Cook, J. A., Burke, J., Mueser, K. T., Pickett-Schenk, S., Grey, D. D., et al. (2005). Clinical factors associated with employment among people with severe mental illness: Findings from the employment intervention demonstration program. Journal of Nervous and Mental Disease, 193(11), 705–713.

    Article  PubMed  Google Scholar 

  • Regenold, M., Sherman, M., & Fenzel, M. (1999). Getting back to work: Self-efficacy as a predictor of employment outcome. Psychiatric Rehabilitation Journal, 22(4), 361–367.

    Google Scholar 

  • Rothwell, P. M. (2005). External validity of randomized controlled trials: ‘‘To whom do the results of this trial apply?’’. Lancet, 365, 82–93.

    Article  PubMed  Google Scholar 

  • Rubin, W. V., & Panzano, P. C. (2002). Identifying meaningful subgroups of adults with severe mental illness. Psychiatric Services, 53(4), 452–457.

    Article  PubMed  Google Scholar 

  • Ruscio, A. M., & Holohan, D. R. (2006). Applying empirically supported treatments to complex cases: Ethical, empirical, and practical considerations. Clinical Psychology: Science and Practice, 13(2), 146–162.

    Article  Google Scholar 

  • Salyers, M. P., McHugo, G. H., Cook, J. A., Razzano, L. A., Drake, R. E., & Mueser, K. T. (2001). Reliability of instruments in a cooperative, multisite study: Employment intervention demonstration program. Mental Health Services Research, 3(3), 129–139.

    Article  PubMed  CAS  Google Scholar 

  • Slade, E., & Salkever, D. (2001). Symptom effects on employment in a structural model of mental illness and treatment: Analysis of patients with schizophrenia. Journal of Mental Health Policy and Economics, 4(1), 25–34.

    PubMed  Google Scholar 

  • SPSS. (1999). Statistical Program for the Social Sciences (Version 11) [Windows]. Chicago.

  • Stein, L. I., & Test, M. A. (1980). Alternative to mental hospital treatment. I. Conceptual model, treatment program, and clinical evaluation. Archives of General Psychiatry, 37(4), 392–397.

    PubMed  CAS  Google Scholar 

  • Stone-Romero, E. F., & Anderson, L. E. (1994). Relative power of moderated multiple regression and the comparison of subgroup correlation coefficients for detecting moderating effects. Journal of Applied Psychology, 79(3), 354–359.

    Article  Google Scholar 

  • Stout, C. E., & Hayes, R. A. (Eds.). (2005). Evidence-based practice: Methods, models, and tools for mental health professionals. New York, NY: John Wiley.

    Google Scholar 

  • Trach, J. S. (1990). Supported employment program characteristics. In R. Rusch (Ed.), Supported employment: Models, methods and issues. Sycamore, IL: Sycamore Press.

    Google Scholar 

  • Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., & Rosenthal, R. (2008). Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine, 358, 252–260.

    Article  PubMed  CAS  Google Scholar 

  • Twamley, E., Jeste, D. V., & Lehman, A. (2003). Vocational rehabilitation in schizophrenia and other psychotic disorders: A literature review and meta-analysis of randomized controlled trials. Journal of Nervous and Mental Disease, 191(8), 515–523.

    Article  PubMed  Google Scholar 

  • Uehara, E., Srebnik, D., & Smukler, M. (2003). Statistical and consensus-based strategies for grouping consumers in mental health level-of-care schemes. Administration and Policy in Mental Health, 30(4), 287–306.

    Article  PubMed  Google Scholar 

  • Ward, J. H. (1963). Hierarchical grouping to optimize an objective function. Journal of the American Statistical Association, 58, 236–244.

    Article  Google Scholar 

  • Wells, K. B. (1999). Treatment research at the crossroads: The scientific interface of clinical trials and effectiveness research. American Journal of Psychiatry, 156(1), 5–10.

    PubMed  CAS  Google Scholar 

  • West, S. G., & Aiken, L. S. (1997). Toward understanding individual effects in multicomponent prevention programs: Design and analysis strategies. In K. J. Bryant, M. Windle, & S. G. West (Eds.), The science of prevention: Methodological advances from alcohol and substance abuse research. Washington, DC: American Psychological Association.

    Google Scholar 

  • Wewiorski, N. J., & Fabian, E. S. (2004). Association between demographic and diagnostic factors and employment outcomes for people with psychiatric disabilities: A synthesis of recent research. Mental Health Services Research, 6(1), 9–21.

    Article  PubMed  Google Scholar 

  • Workforce Investment Act of (1998): Public law 105–220.

Download references

Acknowledgments

This project was funded by an interdisciplinary research grant from the National Institute of Mental Health (R01-MH-62628), and was part of the multi-project Employment Intervention Demonstration Program (EIDP) funded by the Substance Abuse and Mental Health Service Administration (SM-51831). Supplemental support was provided by the van Ameringen Foundation, the John D. and Catherine T. MacArthur Foundation, Massachusetts Department of Mental Health, and the University of Massachusetts Medical School. No author, affiliated institution, or funding organization has a conflict of interest. The authors thank their NIMH project monitor, Ann Hohmann, Ph.D., for her helpful insights and encouragement.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Cathaleene Macias.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Macias, C., Jones, D.R., Hargreaves, W.A. et al. When Programs Benefit Some People More than Others: Tests of Differential Service Effectiveness. Adm Policy Ment Health 35, 283–294 (2008). https://doi.org/10.1007/s10488-008-0174-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-008-0174-y

Keywords

Navigation