Abstract
Background
The practice of evidence-based medicine depends on the availability of clinically relevant research, yet questions have been raised about the generalizability of findings from randomized controlled trials (RCTs).
Objectives
The aim of this study was to quantify differences between RCT patients and treatments and those in day-to-day clinical practice.
Research Design
Data from published reports of two key RCTs underlying recent treatment advances in psychiatry were compared with data on routine psychiatric practice collected through a Practice Research Network (PRN).
Setting
Hospital inpatient units (RCT) and the full range of psychiatric practice settings in the United States (PRN).
Subjects
Adults with bipolar I disorder and adults with schizophrenia.
Measures
Demographic (age, gender, race), clinical (principal diagnoses, comorbid conditions, psychosocial functioning, and histories of hospitalization), and treatment (medication name and dosage) characteristics.
Results
PRN patients had more comorbid conditions and were more likely to be white, female, and older than RCT patients. In all, 38% of PRN patients with schizophrenia and 55% of PRN patients with bipolar I disorder would have been ineligible for the corresponding RCT. Most PRN patients receiving an RCT study medication were also receiving other medications not allowed by the RCT protocol.
Conclusions
Findings support the assertion that RCT patients and treatments are not typical of those in clinical practice, and most patients in clinical practice are receiving treatments that do not have direct empirical support. Research is needed to determine the extent to which RCT findings should be used to guide routine clinical decisions.
Similar content being viewed by others
References
Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS (1996) Evidence based medicine: what it is and what it isn’t. BMJ 312:71–72
Sackett DL, Haynes RB,Guyatt GH,Tugwell P (1991) Clinical Epidemiology: A Basic Science for Clinical Medicine. Boston: Little, Brown and Company, p 281
Starfield B (1990) Quality-of-care research: Internal elegance and external relevance. JAMA 280:1006–1008
Zarin DA, West JC, Pincus HA, McIntyre JS (1995) The American Psychiatric Association Practice Research Network. In: Sederer L, Dickey B, eds. Outcomes Assessment in Clinical Practice. Baltimore, MD: William and Wilkins, pp 146–155
Begg CB (1987) Cancer clinical trials in the USA: Patient eligibility, generalizability of results and technology transfer. Bull Cancer 74:197–203
Chlebowski RT, Butler J, Nelson A, Lillington L (1993) Breast cancer chemoprevention. Tamoxifen: current issues and future prospective. Cancer 72(Suppl 3):1032–1037
Tannock IF (1992) Assessment of study design in clinical trials for bladder cancer. Urol Clin North Am 19:655–662
Lee KL, Califf RM, Simes J, Van de Werf F, Topol EJ (1994) Holding GUSTO up to the light. Global Utilization of Streptokinase and Tissue Plasminogen Activator for Occluded Coronary Arteries. Ann Intern Med 120:876–881
Mulrow CD, Cornell JA, Herrera CR, Kadri A, Farnett L,Aguilar C (1994) Hypertension in the elderly. Implications and generalizability of randomized trials. JAMA 272:1932–1938
Egglin TK, Horowitz RI (1996) The case for better research standards in peripheral thrombolysis: poor quality of randomized trials during the past decade. Acad Radiol 3:1–9
Koch A, Bouges S, Ziegler S, Dinkel H, Daures JP, Victor N (1997) Low molecular weight heparin and unfractionated heparin in thrombosis prophylaxis after major surgical intervention: update of previous meta-analyses. Br J Surg 84:750–759
Saver JL (1995) Coping with an embarrassment of riches. How stroke centers may participate in multiple, concurrent clinical stroke trials. Stroke 26:1289–1292
Schneider LS, Olin JT, Lyness SA, Chui HC (1997) Eligibility of Alzheimer’s disease clinic patients for clinical trials. J Am Geriatr Soc 45:923–928
Cunningham WE, Bozzette SA, Hays RD, Kanouse DE, Shapiro MF (1995) Comparison of health-related quality of life in clinical trials and nonclinical trial human immunodeficiency virus-infected cohorts. Med Care 33(Suppl 4):AS15–AS25
Kramer MS, Shapiro SH (1984) Scientific challenges in the application of randomized trials. JAMA 252:2739–2745
Greil W, Ludwig-Mayerhofer W, Steller B, et al. (1993) The recruitment process for a multicenter study on the long-term prophylactic treatment of affective disorders. J Affect Disord 28:257–265
Licht RW, Gouliaev G,Vestergaard P, Frydenberg M (1997) Generalisability of results from randomised drug trials. Br J Psychiatry 170:264–267
Leber PD, Davis CS (1998) Threats to the validity of clinical trials employing enrichment strategies for sample selection. Control Clin Trials 9:178–187
Miller CA, Hooper CL, Bakish D (1997) A comparison of patients with major depressive disorder recruited through newspaper advertising versus consultation referrals for clinical drug trials. Psychopharmacol Bull 33:69–73
Amori G, Lenox RH (1989) Do volunteer subjects bias clinical trials? J Clin Psychopharmacol 9:321–327
Taylor KM, Feldstein ML, Skeel RT, Pandya KJ, Ng P, Carbone PP (1994) Fundamental dilemmas of the randomized clinical trial process: results of a survey of the 1,737 Eastern Cooperative Oncology Group Investigators. Clin Oncol 12:1796–1805
Antman K, Amato D, Wood W, Carson J, Suit H, Proppe K, Carey R, Greenberger J, Wilson R, Frei E 3rd (1985) Selection bias in clinical trials. Clin Oncol 3:1142–1147
Skerritt U, Pitt B, Armstrong S, O’Brien A (1996) Recruiting patients for drug trials: a difficult task. Psychiatr Bull 20:708–710
Layde PM, Broste SK, Desbiens N, Follen M, Lynn J, Reding D, Vidaillet H (1996) Generalizability of clinical studies conducted at tertiary care medical centers: a population-based analysis. Clin Epidemiol 49:835–841
Smith P, Arnesen H (1998) Non-respondents in a post-myocardial infarction trial: characteristics and reasons for refusal. Acta Med Scand 223:537–542
Torgerson DJ, Sibbald B (1998) Understanding controlled trials: what is a patient preference trial? BMJ 316:360
Llewellyn-Thomas HA, McGreal MJ, Thiel EC, Fine S, Erlichman C (1991) Patients’ willingness to enter clinical trials: measuring the association with perceived benefit and preference for decision participation. Soc Sci Med 32:35–42
Liberati A, Himel HN, Chalmers TC (1986) A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 4:942–951
Edwards AG, Rollnick S (1997) Outcome studies of brief alcohol intervention in general practice: the problem of lost subjects. Addiction 12:1699–1704
Brown C, Schulberg HC (1995) The efficacy of psychosocial treatments in primary care. A review of randomized clinical trials. Gen Hosp Psychiatry 17:414–424
Moncrieff J, Drummond DC (1997) New drug treatments for alcohol problems: a critical appraisal. Addiction 92:939–947
Scott JE, Dixon LB (1995) Psychological interventions for schizophrenia. Schizophr Bull 21:621–630
May WW (1998) Findings from Project MATCH: fact or artifact? Behav Health Management 18:38–39
Fortmann SP, Killen JD (1994) Who shall quit? Comparison of volunteer and population-based recruitment in two minimalcontact smoking cessation studies. Am J Epidemiol 140:39–51
Rapaport MH, Frevert T, Babior S, Zisook S, Judd LL (1995) A comparison of demographic variables, symptom profiles, and measurements of functioning in symptomatic volunteers and an outpatient clinical population. Psychopharmacol Bull 31:111–114
Drummond DC (1997) Alcohol interventions: do the best things come in small packages? Addiction 92:375–379
Essock SM, Hargreaves WA, Covell NH, Goethe J (1996) Clozapine’s effectiveness for patients in state hospitals: results from a randomized trial. Psychopharmacol Bull 32(4):683–697
Zarin DA, Pincus HA, West JC, McIntyre JS (1997) Practice-based research in psychiatry. Am J Psychiatry 154:1199–1208
American Psychiatric Association (1994) Practice guideline for the treatment of patients with bipolar disorder.Am J Psychiatry 151(Suppl):1–36
American Psychiatric Association (1997) Practice guideline for the treatment of patients with schizophrenia. Am J Psychiatry 154(Suppl):1–63
Zarin DA, Seigle L, Pincus HA, MyIntyre JS (1997) Evidence-based practice guidelines. Psychopharmacol Bull 33:641–646
Bowden CL, Brugger AM, Swann AC, Calabrese JR, Janicak PG, Petty F, Dilsaver SC, Davis JM, Rush AJ, Small JG, et al. (1994) Efficacy of divalproex vs lithium and placebo in the treatment of mania. JAMA 271:918–924
Pope HG Jr, McElroy SL, Keck PE Jr, Hudson JI (1991) Valproate in the treatment of acute mania: a placebo controlled study. Arch Gen Psychiatry 48:62–68
Marder SR, Meibach RC (1994) Risperidone in the treatment of schizophrenia. Am J Psychiatry 151:825–835
Zarin DA, Pincus HA, Peterson BD, West JC, Suarez AP, Marcus SC, McIntyre JS (1998) Characterizing psychiatry with findings from the 1996 National Survey of Psychiatric Practice. Am J Psychiatry 155:397–404
West JC, Zarin DA, Peterson BD, Pincus HA (1998) Assessing the feasibility of recruiting a randomly selected sample of psychiatrists to participate in a national practice-based research network. Soc Psychiatry Psychiatr Epidemiol 33:620–623
Pincus HA, Zarin DA, Tanielian TL, Johnson JL, West JC, Pettit AR, Marcus SC, Kessler RC, McIntyre JS (1999) Psychiatric patients and treatments in 1997: findings from the American Psychiatric Practice Research Network. Arch Gen Psychiatry 56(5):441–449
USP DI (1998) Volume I: Drug Information for the Health Care Professional. 18th edition. Rockville, MD: The United States Pharmacopeial Convention, Inc
American Psychiatric Association (1994) Diagnostic and Statistical Manual of Mental Disorders, 4th ed. Washington, D. C.: American Psychiatric Association
Shah BV, Barnwell BG, Bieler GS (1997) SUDAAN User’s Manual, Release 7.5. Research Triangle Park, NC: Research Triangle Institute
Olfson M, Klerman GL, Pincus HA (1993) The roles of psychiatrists in organized outpatient mental health settings. Am J Psychiatry 150:625–631
Lehman AF, Thompson JW, Dixon LB, Scott JE (1995) Schizophrenia: treatment outcomes research—editor’s introduction. Schizophr Bull 21:561–566
Lebowitz BD,Rudorfer MV (1998) Treatment research at the millennium: from efficacy to effectiveness (editorial). J Clin Psychopharmacol 18:1
Stahl SM (1999) Antipsychotic polypharmacy, part 1: Therapeutic option of dirty little secret? J Clin Psychiatry 60:425–426
Kingsbury SJ, Yi D, Simpson GM (2001) Rational and irrational polypharmacy. Psych Services 52:1033–1036
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Zarin, D.A., Young, J.L. & West, J.C. Challenges to evidence-based medicine:. Soc Psychiatry Psychiatr Epidemiol 40, 27–35 (2005). https://doi.org/10.1007/s00127-005-0838-9
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/s00127-005-0838-9