Abstract
Random assignment to a preferred experimental condition can increase service engagement and enhance outcomes, while assignment to a less-preferred condition can discourage service receipt and limit outcome attainment. We examined randomized trials for one prominent psychiatric rehabilitation intervention, supported employment, to gauge how often assignment preference might have complicated the interpretation of findings. Condition descriptions, and greater early attrition from services-as-usual comparison conditions, suggest that many study enrollees favored assignment to new rapid-job-placement supported employment, but no study took this possibility into account. Reviews of trials in other service fields are needed to determine whether this design problem is widespread.
Similar content being viewed by others
References
Abelson, R. P. (1995). Statistics as principled argument. Hillsdale, NJ: Lawrence Erlbaum.
Adair, J. G., Lindsay, R. C. L., & Carlopio, J. (1983). Social artifact research and ethical regulation: Their impact on the teaching of experimental methods. Teaching of Psychology, 10, 159–162. doi:10.1207/s15328023top1003_10.
Aguinis, H. (2004). Regression analysis for categorical moderators. New York: Guilford Press.
Aiken, L. S., & West, S. G. (1991). Multiple regression: Testing and interpreting interactions. Newbury Park, CA: Sage.
Alverson, H., Alverson, M., Drake, R. E., & Becker, D. R. (1998). Social correlates of competitive employment among people with severe mental illness. Psychosocial Rehabilitation Journal, 22(1), 34–40.
Alverson, M., Becker, D. R., & Drake, R. E. (1995). An ethnographic study of coping strategies used by people with severe mental illness participating in supported employment. Psychosocial Rehabilitation Journal, 18(4), 115–127.
Aronson, E. (1999). The power of self-persuasion. The American Psychologist, 54(11), 873–875. doi:10.1037/h0088188.
Beveridge, W. I. B. (1950). The art of scientific investigation. New York: Vintage Books.
Bickman, L. (1987). The functions of program theory. In L. Bickman (Ed.), Using program theory in evaluation. San Francisco: Jossey-Bass.
Bickman, L. (2002). The death of treatment as usual: An excellent first step on a long road. Clinical Psychology: Science and Practice, 9(2), 195–199. doi:10.1093/clipsy/9.2.195.
Bond, G. R., Becker, D. R., Drake, R. E., Rapp, C., Meisler, N., & Lehman, A. F. (2001a). Implementing supported employment as an evidence-based practice. Psychiatric Services, 52(3), 313–322. doi:10.1176/appi.ps.52.3.313.
Bond, G. R., Becker, D. R., Drake, R. E., & Vogler, K. (1997a). A fidelity scale for the individual placement and support model of supported employment. Rehabilitation Counseling Bulletin, 40(4), 265–284.
Bond, G. R., Campbell, K., Evans, L. J., Gervey, R., Pascaris, A., Tice, S., et al. (2002). A scale to measure quality of supported employment for persons with severe mental illness. Journal of Vocational Rehabilitation, 17(4), 239–250.
Bond, G. R., Drake, R., & Becker, D. (2008a). An update on randomized controlled trials of evidence-based supported employment. Psychiatric Rehabilitation Journal, 31(4), 280–290. doi:10.2975/31.4.2008.280.290.
Bond, G. R., Drake, R. E., Mueser, K. T., & Becker, D. R. (1997b). An update on supported employment for people with severe mental illness. Psychiatric Services, 48(3), 335–346.
Bond, G. R., McHugo, G. H., Becker, D., Rapp, C. A., & Whitley, R. (2008b). Fidelity of supported employment: Lessons learned from the national evidence-based practice project. Psychiatric Rehabilitation Journal, 31(4), 300–305. doi:10.2975/31.4.2008.300.305.
Bond, G. R., Salyers, M. P., Roudebush, R. L., Dincin, J., Drake, R. E., Becker, D. R., et al. (2007). A randomized controlled trial comparing two vocational models for persons with severe mental illness. Journal of Consulting and Clinical Psychology, 75(6), 968–982. doi:10.1037/0022-006X.75.6.968.
Bond, G. R., Vogler, K., Resnick, S. G., Evans, L., Drake, R., & Becker, D. (2001b). Dimensions of supported employment: Factor structure of the IPS fidelity scale. Journal of Mental Health, 10(4), 383–393. doi:10.1080/09638230120041146.
Braver, S. L., & Smith, M. C. (1996). Maximizing both external and internal validity in longitudinal true experiments with voluntary treatments: The ‘combined modified’ design. Evaluation and Program Planning, 19, 287–300. doi:10.1016/S0149-7189(96)00029-8.
Brown, T. G., Seraganian, P., Tremblay, J., & Annis, H. (2002). Matching substance abuse aftercare treatments to client characteristics. Addictive Behaviors, 27, 585–604. doi:10.1016/S0306-4603(01)00195-2.
Burns, T., Catty, J., Becker, T., Drake, R. E., Fioritti, A., Knapp, M., et al. (2007). The effectiveness of supported employment for people with severe mental illness: A randomised controlled trial. Lancet, 370, 1146–1152. doi:10.1016/S0140-6736(07)61516-5.
Calsyn, R., Winter, J., & Morse, G. (2000). Do consumers who have a choice of treatment have better outcomes? Community Mental Health Journal, 36(2), 149–160. doi:10.1023/A:1001890210218.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin, 56, 81–105. doi:10.1037/h0046016.
Collins, M. E., Mowbray, C., & Bybee, D. (2000). Characteristics predicting successful outcomes of participants with severe mental illness in supported education. Psychiatric Services, 51(6), 774–780. doi:10.1176/appi.ps.51.6.774.
Cook, J. A. (1999a). Understanding the failure of vocational rehabilitation: What do we need to know and how can we learn it? Journal of Disability Policy Studies, 10(1), 127–132.
Cook, T. D. (1999b). Considering the major arguments against random assignment: An analysis of the intellectual culture surrounding evaluation in American schools of education. Paper presented at the Harvard Faculty Seminar on Experiments in Education, Cambridge, MA.
Cook, T. D., & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues for field settings. Boston: Houghton Mifflin.
Cook, J. A., Carey, M. A., Razzano, L., Burke, J., & Blyler, C. R. (2002). The pioneer: The employment intervention demonstration program. New Directions for Evaluation, 94, 31–44. doi:10.1002/ev.49.
Corrigan, P. W., & Salzer, M. S. (2003). The conflict between random assignment and treatment preference: Implications for internal validity. Evaluation and Program Planning, 26, 109–121. doi:10.1016/S0149-7189(03)00014-4.
Crowther, R. E., Marshall, M., Bond, G. R., & Huxley, P. (2001). Helping people with severe mental illness to obtain work: Systematic review. BMJ: British Medical Journal, 322(7280), 204–208. doi:10.1136/bmj.322.7280.204.
Delucchi, K. L., & Bostrom, A. (2004). Methods for analysis of skewed data distributions in psychiatric clinical studies: Working with many zero values. The American Journal of Psychiatry, 161(7), 1159–1168. doi:10.1176/appi.ajp.161.7.1159.
Drake, R. E., Becker, D. R., & Anthony, W. A. (1994). A research induction group for clients entering a mental health services research project. Hospital & Community Psychiatry, 45(5), 487–489.
Drake, R. E., Becker, D., & Bond, G. R. (2003). Recent research on vocational rehabilitation for persons with severe mental illness. Current Opinion in Psychiatry, 16, 451–455. doi:10.1097/00001504-200307000-00012.
Drake, R. E., McHugo, G. J., Bebout, R. R., Becher, D. R., Harris, M., Bond, G. R., et al. (1999). A randomized clinical trial of supported employment for inner-city patients with severe mental disorders. Archives of General Psychiatry, 56, 627–633. doi:10.1001/archpsyc.56.7.627.
Drake, R. E., McHugo, G. J., Becker, D., Anthony, W. A., & Clark, R. E. (1996). The new Hampshire study of supported employment for people with severe mental illness. Journal of Consulting and Clinical Psychology, 64(2), 391–399. doi:10.1037/0022-006X.64.2.391.
Essock, S. M., Drake, R., Frank, R. G., & McGuire, T. G. (2003). Randomized controlled trials in evidence-based mental health care: Getting the right answer to the right question. Schizophrenia Bulletin, 29(1), 115–123.
Festinger, L. (1957). A theory of cognitive dissonance. Stanford, CA: Stanford University Press.
Gold, P. B., Meisler, N., Santos, A. B., Carnemolla, M. A., Williams, O. H., & Keleher, J. (2006). Randomized trial of supported employment integrated with assertive community treatment for rural adults with severe mental illness. Schizophrenia Bulletin, 32(2), 378–395. doi:10.1093/schbul/sbi056.
Grilo, C. M., Money, R., Barlow, D. H., Goddard, A. W., Gorman, J. M., Hofmann, S. G., et al. (1998). Pretreatment patient factors predicting attrition from a multicenter randomized controlled treatment study for panic disorder. Comprehensive Psychiatry, 39(6), 323–332. doi:10.1016/S0010-440X(98)90043-8.
Halpern, S. D. (2002). Prospective preference assessment: a method to enhance the ethics and efficiency of randomized controlled trials. Controlled Clinical Trials, 23, 274–288. doi:10.1016/S0197-2456(02)00191-5.
Hofmann, S. G., Barlow, D. H., Papp, L. A., Detweiler, M. F., Ray, S. E., Shear, M. K., et al. (1998). Pretreatment attrition in a comparative treatment outcome study on panic disorder. The American Journal of Psychiatry, 155(1), 43–47.
Honey, A. (2000). Psychiatric vocational rehabilitation: Where are the customers’ views. Psychiatric Rehabilitation Journal, 23(3), 270–279.
Kearney, C., & Silverman, W. (1998). A critical review of pharmacotherapy for youth with anxiety disorders: Things are not as they seem. Journal of Anxiety Disorders, 12(2), 83–102. doi:10.1016/S0887-6185(98)00005-X.
Killackey, E., Jackson, H. J., & McGorry, P. D. (2008). Vocational intervention in first-episode psychosis: Individual placement and support versus treatment as usual. The British Journal of Psychiatry, 193, 114–120. doi:10.1192/bjp.bp.107.043109.
King, M., Nazareth, I., Lampe, F., Bower, P., Chandler, M., Morou, M., et al. (2005). Impact of participant and physician intervention preferences on randomized trials: A systematic review. Journal of the American Medical Association, 293(9), 1089–1099. doi:10.1001/jama.293.9.1089.
Krause, M. S., & Howard, K. I. (2003). What random assignment does and does not do. Journal of Clinical Psychology, 59, 751–766. doi:10.1002/jclp.10170.
Krell, H. V., Leuchter, A. F., Morgan, M., Cook, I. A., & Abrams, M. (2004). Subject expectations of treatment effectiveness and outcome of treatment with an experimental antidepressant. The Journal of Clinical Psychiatry, 65(9), 1174–1179.
Lachenbruch, P. A. (2002). Analysis of data with excess zeros. Statistical Methods in Medical Research, 11, 297–302. doi:10.1191/0962280202sm289ra.
Laengle, G., Welte, W., Roesger, U., Guenthner, A., & U’Ren, R. (2000). Chronic psychiatric patients without psychiatric care: A pilot study. Social Psychiatry and Psychiatric Epidemiology, 35(10), 457–462. doi:10.1007/s001270050264.
Lambert, M. F., & Wood, J. (2000). Incorporating patient preferences into randomized trials. Journal of Clinical Epidemiology, 53, 163–166. doi:10.1016/S0895-4356(99)00146-8.
Latimer, E. A., LeCompte, M. D., Becker, D. R., Drake, R. E., Duclos, I., Piat, M., et al. (2006). Generalizability of the individual placement and support model of supported employment: Results of a canadian randomised controlled trial. The British Journal of Psychiatry, 189, 65–73. doi:10.1192/bjp.bp.105.012641.
Lavori, P. W. (2000). Placebo control groups in randomized treatment trials: A statistician’s perspective. Biological Psychiatry, 47, 717–723. doi:10.1016/S0006-3223(00)00838-6.
Lavori, P. W., Rush, A. J., Wisniewski, S. R., Alpert, J., Fava, M., Kupfer, D. J., et al. (2001). Strengthening clinical effectiveness trials: Equipoise-stratified randomization. Biological Psychiatry, 50(10), 792–801. doi:10.1016/S0006-3223(01)01223-9.
Lehman, A. F., Goldberg, R. W., Dixon, L. B., NcNary, S., Postrado, L., Hackman, A., et al. (2002). Improving employment outcomes for persons with severe mental illnesses. Archives of General Psychiatry, 59(2), 165–172. doi:10.1001/archpsyc.59.2.165.
Lewin, K. (1943). Forces behind food habits and methods of change. Bulletin of the National Research Council, 108, 35–65.
Leykin, Y., DeRubeis, R. J., Gallop, R., Amsterdam, J. D., Shelton, R. C., & Hollon, S. D. (2007). The relation of patients’ treatment preferences to outcome in a randomized clinical trial. Behavior Therapy, 38, 209–217. doi:10.1016/j.beth.2006.08.002.
Lilford, R., & Jackson, J. (1995). Equipoise and the ethics of randomisation. Journal of the Royal Society of Medicine, 88, 552–559.
Little, R. J., & Rubin, D. B. (2000). Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and analytical approaches. Annual Review of Public Health, 21, 121–145. doi:10.1146/annurev.publhealth.21.1.121.
Macias, C., Aronson, E., Hargreaves, W., Weary, G., Barreira, P., Harvey, J.H., et al. (2009). Transforming dissatisfaction with services into self-determination: A social psychological perspective on community program effectiveness. Journal of Applied Social Psychology, 39(7).
Macias, C., Barreira, P., Hargreaves, W., Bickman, L., Fisher, W. H., & Aronson, E. (2005). Impact of referral source and study applicants’ preference for randomly assigned service on research enrollment, service engagement, and evaluative outcomes. The American Journal of Psychiatry, 162(4), 781–787. doi:10.1176/appi.ajp.162.4.781.
Macias, C., Rodican, C. F., Hargreaves, W. A., Jones, D. R., Barreira, P. J., & Wang, Q. (2006). Supported employment outcomes of a randomized controlled trial of assertive community treatment and clubhouse models. Psychiatric Services, 57(10), 1406–1415. doi:10.1176/appi.ps.57.10.1406.
Magidson, J. (2000). On models used to adjust for preexisting differences. In L. Bickman (Ed.), Research design (Vol. 2). Thousand Oaks, CA: Sage.
Marcus, S. M. (1997). Assessing non-consent bias with parallel randomized and nonrandomized clinical trials. Journal of Clinical Epidemiology, 50(7), 823–828. doi:10.1016/S0895-4356(97)00068-1.
Mark, M. M., Hofmann, D. A., & Reichardt, C. S. (1992). Testing theories in theory-driven evaluations: Tests of moderation in all things. In H. Chen & P. H. Rossi (Eds.), Using theory to improve program and policy evaluations. New York: Greenwood Press.
McGrew, J. H., & Griss, M. E. (2005). Concurrent and predictive validity of two scales to assess the fidelity of implementation of supported employment. Psychiatric Rehabilitation Journal, 29(1), 41–47. doi:10.2975/29.2005.41.47.
McGrew, J. H., Pescosolido, B. A., & Wright, E. (2003). Case managers’ perspectives on critical ingredients of Assertive Community Treatment and on its implementation. Psychiatric Services, 54(3), 370–376. doi:10.1176/appi.ps.54.3.370.
McGurk, S., Mueser, K., Feldman, K., Wolfe, R., & Pascaris, A. (2007). Cognitive training for supported employment: 2–3 year outcomes of a randomized controlled trial. The American Journal of Psychiatry, 164, 437–441. doi:10.1176/appi.ajp.164.3.437.
McHugo, G. J., Bebout, R. R., Harris, M., Cleghorn, S., Herring, G., Xie, H., et al. (2004). A randomized controlled trial of integrated versus parallel housing services for homeless adults with severe mental illness. Schizophrenia Bulletin, 30(4), 969–982.
McQuilken, M., Zahniser, J. H., Novak, J., Starks, R. D., Olmos, A., & Bond, G. R. (2003). The work project survey: Consumer perspectives on work. Journal of Vocational Rehabilitation, 18, 59–68.
Meyer, B., Pilkonis, P. A., Krupnick, J. L., Egan, M. K., Simmens, S. J., & Sotsky, S. M. (2002). Treatment expectancies, patient alliance and outcome: Further analyses from the national institute of mental health treatment of depression collaborative research program. Journal of Consulting and Clinical Psychology, 70(4), 1051–1055. doi:10.1037/0022-006X.70.4.1051.
Mowbray, C. T., Collins, M., & Deborah, B. (1999). Supported education for individuals with psychiatric disabilities: Long-term outcomes from an experimental study. Social Work Research, 23(2), 89–100.
Mueser, K. T., Clark, R. E., Haines, M., Drake, R. E., McHugo, G. J., GR, B., et al. (2004). The Hartford study of supported employment for persons with severe mental illness. Journal of Consulting and Clinical Psychology, 72(3), 479–490. doi:10.1037/0022-006X.72.3.479.
Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: Investigating participant demand characteristics. The Journal of General Psychology, 135(2), 151–165. doi:10.3200/GENP.135.2.151-166.
Noble, J.H., Honberg, R.S., Hall, L.L., & Flynn, L.M. (1997). A legacy of failure: The inability of the federal-state vocational rehabilitation system to serve people with severe mental illness. Arlington: VA: National Alliance for the Mentally Ill.
Quimby, E., Drake, R., & Becker, D. (2001). Ethnographic findings from the Washington, DC vocational services study. Psychiatric Rehabilitation Journal, 24(4), 368–374.
Rosenheck, R. A. (2001). Organizational process: A missing link between research and practice. Psychiatric Services, 52(12), 1607–1612. doi:10.1176/appi.ps.52.12.1607.
Schwartz, C. E., & Sprangers, M. (1999). Methodological approaches for assessing response shift in longitudinal quality of life research. Social Science & Medicine, 48, 1531–1548. doi:10.1016/S0277-9536(99)00047-7.
Sechrest, L., Davis, M., Stickle, T., & McKnight, P. (2000). Understanding ‘method’ variance. In L. Bickman (Ed.), Research design. Thousand Oaks, CA: Sage.
Secker, J., Membrey, H., Grove, B., & Seebohm, P. (2002). Recovering from illness or recovering your life? Implications of clinical versus social models of recovery from mental health problems for employment support services. Disability & Society, 17(4), 403–418. doi:10.1080/09687590220140340.
Shadish, W. R. (2002). Revisiting field experimentation: Field notes for the future. Psychological Methods, 7(1), 3–18. doi:10.1037/1082-989X.7.1.3.
Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin.
Shadish, W. R., Matt, G. E., Navarro, A. M., & Phillips, G. (2000). The effects of psychological therapies under clinically representative conditions: A meta-analysis. Psychological Bulletin, 126(4), 512–529. doi:10.1037/0033-2909.126.4.512.
Shapiro, S. L., Figueredo, A. J., Caspi, O., Schwartz, G. E., Bootzin, R. R., Lopez, A. M., et al. (2002). Going quasi: The premature disclosure effect in a randomized clinical trial. Journal of Behavioral Medicine, 25(6), 605–621. doi:10.1023/A:1020693417427.
Sneed, J. R., Rutherford, B. R., Rindskopf, D., Lane, D. T., Sackeim, H. A., & Roose, S. P. (2008). Design makes a difference: A meta-analysis of antidepressant response rates in placebo-controlled versus comparator trials in late-life depression. The American Journal of Geriatric Psychiatry, 16, 65–73. doi:10.1097/JGP.0b013e3181256b1d.
Sosin, M. R. (2002). Outcomes and sample selection: The case of a homelessness and substance abuse intervention. The British Journal of Mathematical and Statistical Psychology, 55(1), 63–92. doi:10.1348/000711002159707.
Staines, G., McKendrick, K., Perlis, T., Sacks, S., & DeLeon, G. (1999). Sequential assignment and treatment as usual: Alternatives to standard experimental designs in field studies of treatment efficacy. Evaluation Review, 23(1), 47–76. doi:10.1177/0193841X9902300103.
TenHave, T., Coyne, J., Salzer, M., & Katz, I. (2003). Research to improve the quality of care for depression: Alternatives to the simple randomized clinical trial. General Hospital Psychiatry, 25, 115–123. doi:10.1016/S0163-8343(02)00275-X.
Torgerson, D., Klaber Moffett, J. A., & Russell, I. T. (1996). Including patient preferences in randomized clinical trials. Journal of Health Services Research & Policy, 1, 194–197.
Torgerson, D., & Moffett, J.K. (2005). Patient Preference and Validity of Randomized Controlled Trials: Letter to the Editor. Journal of the American Medical Association, 294(1), 41. doi:10.1001/jama.294.1.41-b.
Trist, E., & Sofer, C. (1959). Exploration in group relations. Leicester: Leicester University Press.
Twamley, E. W., Narvaez, J. M., Becker, D. R., Bartels, S. J., & Jeste, D. V. (2008). Supported employment for middle-aged and older people with schizophrenia. American Journal of Psychiatric Rehabilitation, 11(1), 76–89. doi:10.1080/15487760701853326.
Wahlbeck, K., Tuunainen, A., Ahokas, A., & Leucht, S. (2001). Dropout rates in randomised antipsychotic drug trials. Psychopharmacology, 155(3), 230–233. doi:10.1007/s002130100711.
West, S. G., Aiken, L. S., & Todd, M. (2002). Probing the effects of individual components in multiple component prevention programs. In T. Revenson & R. B. D’Agostino (Eds.), Ecological research to promote social change: Methodological advances from community psychology. New York, NY: Kluwer.
West, S. G., & Sagarin, B. J. (2000). Participant selection and loss in randomized experiments. In L. Bickman (Ed.), Research design: Donald Campbell’s legacy (Vol. II, pp. 117–154). Thousand Oaks, CA: Sage.
Wolf, J., Coba, C., & Cirella, M. (2001). Education as psychosocial rehabilitation: Supported education program partnerships with mental health and behavioral healthcare certificate programs. Psychiatric Rehabilitation Skills, 5(3), 455–476.
Wong, K.K., Chiu, R., Tang, B., Mak, D., Liu, J., & Chiu, S.N. (2008). A randomized controlled trial of a supported employment program for persons with long-term mental illness in Hong Kong. Psychiatric Services (Washington, D.C.), 59(1), 84–90. doi:10.1176/appi.ps.59.1.84.
Acknowledgements
Work on this article was funded by National Institute of Mental Health grants to the first and second authors (MH62628; MH01903). We are indebted to Ann Hohmann, Ph.D. for her supportive monitoring of the NIMH research grant that fostered this interdisciplinary collaboration, and to anonymous reviewers who offered invaluable insights during manuscript preparation.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Macias, C., Gold, P.B., Hargreaves, W.A. et al. Preference in Random Assignment: Implications for the Interpretation of Randomized Trials. Adm Policy Ment Health 36, 331–342 (2009). https://doi.org/10.1007/s10488-009-0224-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-009-0224-0