Social Psychiatry and Psychiatric Epidemiology

, Volume 40, Issue 1, pp 27–35

Challenges to evidence-based medicine:

A comparison of patients and treatments in randomized controlled trials with patients and treatments in a practice research network
  • Deborah A. Zarin
  • Julia L. Young
  • Joyce C. West



The practice of evidence-based medicine depends on the availability of clinically relevant research, yet questions have been raised about the generalizability of findings from randomized controlled trials (RCTs).


The aim of this study was to quantify differences between RCT patients and treatments and those in day-to-day clinical practice.

Research Design

Data from published reports of two key RCTs underlying recent treatment advances in psychiatry were compared with data on routine psychiatric practice collected through a Practice Research Network (PRN).


Hospital inpatient units (RCT) and the full range of psychiatric practice settings in the United States (PRN).


Adults with bipolar I disorder and adults with schizophrenia.


Demographic (age, gender, race), clinical (principal diagnoses, comorbid conditions, psychosocial functioning, and histories of hospitalization), and treatment (medication name and dosage) characteristics.


PRN patients had more comorbid conditions and were more likely to be white, female, and older than RCT patients. In all, 38% of PRN patients with schizophrenia and 55% of PRN patients with bipolar I disorder would have been ineligible for the corresponding RCT. Most PRN patients receiving an RCT study medication were also receiving other medications not allowed by the RCT protocol.


Findings support the assertion that RCT patients and treatments are not typical of those in clinical practice, and most patients in clinical practice are receiving treatments that do not have direct empirical support. Research is needed to determine the extent to which RCT findings should be used to guide routine clinical decisions.

Key words

evidence-based medicine randomized controlled trials generalizability schizophrenia bipolar disorder 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS (1996) Evidence based medicine: what it is and what it isn’t. BMJ 312:71–72PubMedGoogle Scholar
  2. 2.
    Sackett DL, Haynes RB,Guyatt GH,Tugwell P (1991) Clinical Epidemiology: A Basic Science for Clinical Medicine. Boston: Little, Brown and Company, p 281Google Scholar
  3. 3.
    Starfield B (1990) Quality-of-care research: Internal elegance and external relevance. JAMA 280:1006–1008CrossRefGoogle Scholar
  4. 4.
    Zarin DA, West JC, Pincus HA, McIntyre JS (1995) The American Psychiatric Association Practice Research Network. In: Sederer L, Dickey B, eds. Outcomes Assessment in Clinical Practice. Baltimore, MD: William and Wilkins, pp 146–155Google Scholar
  5. 5.
    Begg CB (1987) Cancer clinical trials in the USA: Patient eligibility, generalizability of results and technology transfer. Bull Cancer 74:197–203Google Scholar
  6. 6.
    Chlebowski RT, Butler J, Nelson A, Lillington L (1993) Breast cancer chemoprevention. Tamoxifen: current issues and future prospective. Cancer 72(Suppl 3):1032–1037Google Scholar
  7. 7.
    Tannock IF (1992) Assessment of study design in clinical trials for bladder cancer. Urol Clin North Am 19:655–662Google Scholar
  8. 8.
    Lee KL, Califf RM, Simes J, Van de Werf F, Topol EJ (1994) Holding GUSTO up to the light. Global Utilization of Streptokinase and Tissue Plasminogen Activator for Occluded Coronary Arteries. Ann Intern Med 120:876–881Google Scholar
  9. 9.
    Mulrow CD, Cornell JA, Herrera CR, Kadri A, Farnett L,Aguilar C (1994) Hypertension in the elderly. Implications and generalizability of randomized trials. JAMA 272:1932–1938CrossRefGoogle Scholar
  10. 10.
    Egglin TK, Horowitz RI (1996) The case for better research standards in peripheral thrombolysis: poor quality of randomized trials during the past decade. Acad Radiol 3:1–9Google Scholar
  11. 11.
    Koch A, Bouges S, Ziegler S, Dinkel H, Daures JP, Victor N (1997) Low molecular weight heparin and unfractionated heparin in thrombosis prophylaxis after major surgical intervention: update of previous meta-analyses. Br J Surg 84:750–759CrossRefGoogle Scholar
  12. 12.
    Saver JL (1995) Coping with an embarrassment of riches. How stroke centers may participate in multiple, concurrent clinical stroke trials. Stroke 26:1289–1292Google Scholar
  13. 13.
    Schneider LS, Olin JT, Lyness SA, Chui HC (1997) Eligibility of Alzheimer’s disease clinic patients for clinical trials. J Am Geriatr Soc 45:923–928Google Scholar
  14. 14.
    Cunningham WE, Bozzette SA, Hays RD, Kanouse DE, Shapiro MF (1995) Comparison of health-related quality of life in clinical trials and nonclinical trial human immunodeficiency virus-infected cohorts. Med Care 33(Suppl 4):AS15–AS25Google Scholar
  15. 15.
    Kramer MS, Shapiro SH (1984) Scientific challenges in the application of randomized trials. JAMA 252:2739–2745CrossRefGoogle Scholar
  16. 16.
    Greil W, Ludwig-Mayerhofer W, Steller B, et al. (1993) The recruitment process for a multicenter study on the long-term prophylactic treatment of affective disorders. J Affect Disord 28:257–265CrossRefGoogle Scholar
  17. 17.
    Licht RW, Gouliaev G,Vestergaard P, Frydenberg M (1997) Generalisability of results from randomised drug trials. Br J Psychiatry 170:264–267Google Scholar
  18. 18.
    Leber PD, Davis CS (1998) Threats to the validity of clinical trials employing enrichment strategies for sample selection. Control Clin Trials 9:178–187CrossRefGoogle Scholar
  19. 19.
    Miller CA, Hooper CL, Bakish D (1997) A comparison of patients with major depressive disorder recruited through newspaper advertising versus consultation referrals for clinical drug trials. Psychopharmacol Bull 33:69–73Google Scholar
  20. 20.
    Amori G, Lenox RH (1989) Do volunteer subjects bias clinical trials? J Clin Psychopharmacol 9:321–327Google Scholar
  21. 21.
    Taylor KM, Feldstein ML, Skeel RT, Pandya KJ, Ng P, Carbone PP (1994) Fundamental dilemmas of the randomized clinical trial process: results of a survey of the 1,737 Eastern Cooperative Oncology Group Investigators. Clin Oncol 12:1796–1805Google Scholar
  22. 22.
    Antman K, Amato D, Wood W, Carson J, Suit H, Proppe K, Carey R, Greenberger J, Wilson R, Frei E 3rd (1985) Selection bias in clinical trials. Clin Oncol 3:1142–1147Google Scholar
  23. 23.
    Skerritt U, Pitt B, Armstrong S, O’Brien A (1996) Recruiting patients for drug trials: a difficult task. Psychiatr Bull 20:708–710Google Scholar
  24. 24.
    Layde PM, Broste SK, Desbiens N, Follen M, Lynn J, Reding D, Vidaillet H (1996) Generalizability of clinical studies conducted at tertiary care medical centers: a population-based analysis. Clin Epidemiol 49:835–841CrossRefGoogle Scholar
  25. 25.
    Smith P, Arnesen H (1998) Non-respondents in a post-myocardial infarction trial: characteristics and reasons for refusal. Acta Med Scand 223:537–542Google Scholar
  26. 26.
    Torgerson DJ, Sibbald B (1998) Understanding controlled trials: what is a patient preference trial? BMJ 316:360Google Scholar
  27. 27.
    Llewellyn-Thomas HA, McGreal MJ, Thiel EC, Fine S, Erlichman C (1991) Patients’ willingness to enter clinical trials: measuring the association with perceived benefit and preference for decision participation. Soc Sci Med 32:35–42CrossRefGoogle Scholar
  28. 28.
    Liberati A, Himel HN, Chalmers TC (1986) A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 4:942–951Google Scholar
  29. 29.
    Edwards AG, Rollnick S (1997) Outcome studies of brief alcohol intervention in general practice: the problem of lost subjects. Addiction 12:1699–1704CrossRefGoogle Scholar
  30. 30.
    Brown C, Schulberg HC (1995) The efficacy of psychosocial treatments in primary care. A review of randomized clinical trials. Gen Hosp Psychiatry 17:414–424CrossRefGoogle Scholar
  31. 31.
    Moncrieff J, Drummond DC (1997) New drug treatments for alcohol problems: a critical appraisal. Addiction 92:939–947CrossRefGoogle Scholar
  32. 32.
    Scott JE, Dixon LB (1995) Psychological interventions for schizophrenia. Schizophr Bull 21:621–630Google Scholar
  33. 33.
    May WW (1998) Findings from Project MATCH: fact or artifact? Behav Health Management 18:38–39Google Scholar
  34. 34.
    Fortmann SP, Killen JD (1994) Who shall quit? Comparison of volunteer and population-based recruitment in two minimalcontact smoking cessation studies. Am J Epidemiol 140:39–51Google Scholar
  35. 35.
    Rapaport MH, Frevert T, Babior S, Zisook S, Judd LL (1995) A comparison of demographic variables, symptom profiles, and measurements of functioning in symptomatic volunteers and an outpatient clinical population. Psychopharmacol Bull 31:111–114Google Scholar
  36. 36.
    Drummond DC (1997) Alcohol interventions: do the best things come in small packages? Addiction 92:375–379CrossRefPubMedGoogle Scholar
  37. 37.
    Essock SM, Hargreaves WA, Covell NH, Goethe J (1996) Clozapine’s effectiveness for patients in state hospitals: results from a randomized trial. Psychopharmacol Bull 32(4):683–697Google Scholar
  38. 38.
    Zarin DA, Pincus HA, West JC, McIntyre JS (1997) Practice-based research in psychiatry. Am J Psychiatry 154:1199–1208Google Scholar
  39. 39.
    American Psychiatric Association (1994) Practice guideline for the treatment of patients with bipolar disorder.Am J Psychiatry 151(Suppl):1–36 Google Scholar
  40. 40.
    American Psychiatric Association (1997) Practice guideline for the treatment of patients with schizophrenia. Am J Psychiatry 154(Suppl):1–63Google Scholar
  41. 41.
    Zarin DA, Seigle L, Pincus HA, MyIntyre JS (1997) Evidence-based practice guidelines. Psychopharmacol Bull 33:641–646Google Scholar
  42. 42.
    Bowden CL, Brugger AM, Swann AC, Calabrese JR, Janicak PG, Petty F, Dilsaver SC, Davis JM, Rush AJ, Small JG, et al. (1994) Efficacy of divalproex vs lithium and placebo in the treatment of mania. JAMA 271:918–924CrossRefGoogle Scholar
  43. 43.
    Pope HG Jr, McElroy SL, Keck PE Jr, Hudson JI (1991) Valproate in the treatment of acute mania: a placebo controlled study. Arch Gen Psychiatry 48:62–68Google Scholar
  44. 44.
    Marder SR, Meibach RC (1994) Risperidone in the treatment of schizophrenia. Am J Psychiatry 151:825–835PubMedGoogle Scholar
  45. 45.
    Zarin DA, Pincus HA, Peterson BD, West JC, Suarez AP, Marcus SC, McIntyre JS (1998) Characterizing psychiatry with findings from the 1996 National Survey of Psychiatric Practice. Am J Psychiatry 155:397–404Google Scholar
  46. 46.
    West JC, Zarin DA, Peterson BD, Pincus HA (1998) Assessing the feasibility of recruiting a randomly selected sample of psychiatrists to participate in a national practice-based research network. Soc Psychiatry Psychiatr Epidemiol 33:620–623CrossRefGoogle Scholar
  47. 47.
    Pincus HA, Zarin DA, Tanielian TL, Johnson JL, West JC, Pettit AR, Marcus SC, Kessler RC, McIntyre JS (1999) Psychiatric patients and treatments in 1997: findings from the American Psychiatric Practice Research Network. Arch Gen Psychiatry 56(5):441–449CrossRefGoogle Scholar
  48. 48.
    USP DI (1998) Volume I: Drug Information for the Health Care Professional. 18th edition. Rockville, MD: The United States Pharmacopeial Convention, IncGoogle Scholar
  49. 49.
    American Psychiatric Association (1994) Diagnostic and Statistical Manual of Mental Disorders, 4th ed. Washington, D. C.: American Psychiatric AssociationGoogle Scholar
  50. 50.
    Shah BV, Barnwell BG, Bieler GS (1997) SUDAAN User’s Manual, Release 7.5. Research Triangle Park, NC: Research Triangle InstituteGoogle Scholar
  51. 51.
    Olfson M, Klerman GL, Pincus HA (1993) The roles of psychiatrists in organized outpatient mental health settings. Am J Psychiatry 150:625–631Google Scholar
  52. 52.
    Lehman AF, Thompson JW, Dixon LB, Scott JE (1995) Schizophrenia: treatment outcomes research—editor’s introduction. Schizophr Bull 21:561–566Google Scholar
  53. 53.
    Lebowitz BD,Rudorfer MV (1998) Treatment research at the millennium: from efficacy to effectiveness (editorial). J Clin Psychopharmacol 18:1CrossRefGoogle Scholar
  54. 54.
    Stahl SM (1999) Antipsychotic polypharmacy, part 1: Therapeutic option of dirty little secret? J Clin Psychiatry 60:425–426PubMedGoogle Scholar
  55. 55.
    Kingsbury SJ, Yi D, Simpson GM (2001) Rational and irrational polypharmacy. Psych Services 52:1033–1036CrossRefGoogle Scholar

Copyright information

© Steinkopff Verlag 2005

Authors and Affiliations

  • Deborah A. Zarin
    • 2
  • Julia L. Young
    • 3
  • Joyce C. West
    • 1
  1. 1.American Psychiatric Institute for Research and Education Practice Research NetworkArlington (VA)USA
  2. 2.Technology Assessment Program Agency for Healthcare Research and Quality (AHRQ)Rockville (MD)USA
  3. 3.Mathematica Policy Research, Inc.Washington (D.C.)USA

Personalised recommendations