Skip to main content
SpringerLink
Log in

What Is the Evidence for Evidence-Based Treatments? A Hard Look at Our Soft Underbelly

  • Published:
Mental Health Services Research

Abstract

In the rising quest for evidence-based interventions, recent research often does not give adequate attention to “nonspecific therapeutic factors,” including the effects of attention, positive regard, and therapeutic alliance, as well as the effects of treatment dose, intensity and actual processes mediating therapeutic change. To determine the extent to which recent clinical trial designs fully this problem, the authors conducted a systematic review of PsychLit/Medline of all controlled child psychotherapy treatment studies from 1995 to 2004. A total of 52 studies were identified that met review criteria: two or more therapy conditions and random assignment of participants to intervention groups. Of the 52 studies, one group (n = 29) compared a presumably active treatment with 1 or more similarly intensive treatments (often an attention control group) presumably not containing the active therapeutic ingredients. Of these, 14 studies found evidence of consistent differences between the two groups, whereas 15 did not. An additional group of studies (n = 27) compared therapy groups with different levels of intensity and “dose” of the putatively active treatment; 13 of these found evidence of the effects of different levels of treatment dose/intensity on outcomes and 14 did not. Four studies met criteria for inclusion in both groups. Across both groups of studies, when positive effects were found, few studies systematically explored whether the presumed active therapeutic ingredients actually accounted for the degree of change, nor did they often address plausible alternative explanations, such as nonspecific therapeutic factors of positive expectancies, therapeutic alliance, or attention. Findings suggest that many child psychotherapy treatment studies have not inadequately controlled for nonspecific factors such as attention and treatment intensity and have failed to assess specific mediators of change. Specific recommendations for future studies are offered, specifically:

  1. 1.

    Initial specification in study design how investigators will test if the intervention is efficacious over and above the effects of positive expectancies, positive regard, or attention;

  2. 2.

    Planned (rather than post hoc) analyses to explore whether and how a given treatment’s specific ingredients or the overall intensity of its “dose” are related to treatment outcomes;

  3. 3.

    More studies conducting head-to-head tests of different types but equally credible forms of treatment, with planned analyses testing different mechanisms of change

  4. 4.

    When positive effects of a treatment vs. a control are found, systematic elimination of specific, competing hypotheses of reasons for treatment efficacy (e.g., attention, therapeutic alliance, face validity of treatment and client’s treatment expectations, changes in self-efficacy), and

  5. 5.

    When negative effects are reported, appropriate analyses to address alternative explanations (lack of power, floor or ceiling effects, mediator analyses to address possible fidelity or adherence problems, therapist effects, absence of main effects but possible subgroup/moderator effects, etc.).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abikoff, H., Hechtman, L., Klein, R. G., Weiss, G., Fleiss, K., Etcovitch, J., et al. (2004). Symptomatic improvement in children with ADHD treated with long-term methylphenidate and multimodal psychosocial treatment. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 802–811.

    Google Scholar 

  • Barrett, P. M. (1998). Evaluation of cognitive-behavioral group treatments for childhood anxiety disorders. Journal of Clinical Child Psychology, 27, 459–468.

    Google Scholar 

  • Barrett, P., Healy-Farrell, L., & March, J. S. (2004). Cognitive-behavioral family treatment of childhood obsessive-compulsive disorder: A controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 46–62.

    Google Scholar 

  • Barrett, P. M., Dadds, M. R., & Rapee, R. M. (1996). Family treatment of childhood anxiety: A controlled trial. Journal of Consulting and Clinical Psychology, 64, 333–342.

    Google Scholar 

  • Baskin, T. W., Tierney, S. C., Minami, T., & Wampold, B. E. (2003). Establishing specificity in psychotherapy: A meta-analysis of structural equivalence of placebo controls. Journal of Consulting and Clinical Psychology, 71, 973–979.

    Google Scholar 

  • Beidel, D. C., Turner, S. M., & Morris, T. L. (2000). Behavioral treatment of childhood social phobia. Journal of Consulting and Clinical Psychology, 68, 1072–1080.

    Google Scholar 

  • Bickman, L., Heflinger, C. A., Northrup, D., Sonnichsen, S., & Schilling, S. (1998). Long-term outcomes of family caregiver empowerment. Journal of Child and Family Studies, 7, 269–282.

    Google Scholar 

  • Birmaher, B., Brent, D. A., Kolko, D., Baugher, M., Bridge, J., Holder, D., et al. (2000), Clinical outcome after short-term psychotherapy for adolescents with major depressive disorder. Archives of General Psychiatry, 57, 29–36.

    Google Scholar 

  • Brent, D. A., Holder, D., Kolko, D., Birmaher, B., Baugher, M., Roth, C., et al. (1997). A clinical psychotherapy trial for adolescent depression comparing cognitive, family, and supportive therapy. Archives of General Psychiatry, 54, 877–885.

    Google Scholar 

  • Chadwick, O., Momcilovic, N., Rossiter, R., Stumbles, E., & Taylor, E. (2001). A randomized trial of brief individual versus group parent training for behaviour problems in children with severe learning disabilities. Behavioural and Cognitive Psychotherapy, 29, 151–167.

    Google Scholar 

  • Chamberlain, P., & Reid, J. B. (1998). Comparison of two community alternatives to incarceration for chronic juvenile offenders. Journal of Consulting and Clinical Psychology, 66, 624–633.

    Google Scholar 

  • Chambless, D., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7–18.

    Google Scholar 

  • Clarke, G. N., Hornbrook, M., Lynch, F., & Polen, M. (2002). Group cognitive-behavioral treatment for depressed adolescent offspring of depressed parents in a health maintenance organization. Journal of the American Academy of Child and Adolescent Psychiatry, 41, 305–313.

    Google Scholar 

  • Clarke, G. N., Rohde, P., Lewinsohn, P. M., Hops, H., & Seeley, J. R. (1999). Cognitive-behavioral treatment of adolescent depression: Efficacy of acute group treatment and booster sessions. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 272–279.

    Google Scholar 

  • Cobham, V. E., Dadds, M. R., & Spence, S. H. (1998). The role of parental anxiety in the treatment of childhood anxiety. Journal of Consulting and Clinical Psychology, 66, 893–905.

    Google Scholar 

  • Cohen, J. (1988). Statistical power analyses for the behavioral sciences (2nd edn.). Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Cohen, J. A., & Mannarino, A. P. (1996). A treatment outcome study for sexually abused preschool children. Journal of the American Academy of Child and Adolescent Psychiatry, 35, 42–50.

    Google Scholar 

  • Cohen, N. J., Muir, E., Parker, C. J., Brown, M., Lojkasek, M., Muir, R., et al. (1999). Watch, wait and wonder: Testing the effectiveness of a new approach to mother-infant psychotherapy. Infant Mental Health Journal, 20, 429–451.

    Google Scholar 

  • Connolly, L., Sharry, J., & Fitzpatrick, C. (2001). Evaluation of a group treatment programme for parents of children with behavioural disorders. Child and Adolescent Mental Health, 6, 159–165.

    Google Scholar 

  • Dadds, M. R., Spence, S. H., Holland, D. E., Barrett, P. M., & Laurens, K. R. (1997). Prevention and early intervention for anxiety disorders: A controlled trial. Journal of Consulting and Clinical Psychology, 65, 627–635.

    Google Scholar 

  • Dishion, T. J., & Andrews, D. W. (1995). Preventing escalation in problem behaviors with high-risk young adolescents: immediate and 1-year outcomes. Journal of Consulting and Clinical Psychology, 63, 538–548

    Google Scholar 

  • dos Santos, E. L. C., Marturano, E. M., de Almeida Motta, A. M., & Giurlana, A. G. (2003). Treating boys with low school achievement and behavior problems: Comparison of two kinds of intervention. Psychological Reports, 92, 105–116.

    Google Scholar 

  • Driskill, J. D. (2000). Structured child and parent groups with ADHD children: Evaluation of varying levels of parent involvement. Dissertation Abstracts International: B, 60(9–B), 4884.

    Google Scholar 

  • Eddy, J. M., & Chamberlain, P. (2000). Family management and deviant peer association as mediators of the impact of treatment condition on youth antisocial behavior. Journal of Consulting and Clinical Psychology, 68, 857–863.

    Google Scholar 

  • Eddy, J. M., Whaley, R. B., & Chamberlain, P. (2004). The prevention of violent behavior by chronic and serious male juvenile offenders: A 2-year follow-up of a randomized clinical trial. Journal of Emotional and Behavioral Disorders, 12, 2–8.

    Google Scholar 

  • Eisler, I., Dare, C., Hodes, M., Russell, G., Dodge, E., & Le Grange, D. (2000). Family therapy for adolescent anorexia nervosa: The results of a controlled comparison of two family interventions. Journal of Child Psychology and Psychiatry and Allied Disciplines, 41, 727–736.

    Google Scholar 

  • Flannery-Schroeder, E. C., & Kendall, P. C. (2000). Group and individual cognitive-behavioral treatments for youth with anxiety disorders: A randomized clinical trial. Cognitive Therapy and Research, 24, 251–278.

    Google Scholar 

  • Forgatch, M. S., & DeGarmo, D. S. (1999). Parenting through change: An effective prevention program for single mothers. Journal of Consulting and Clinical Psychology, 67, 711–724.

    Google Scholar 

  • Hebert, D. A. (2002). Evaluating cognitive-behavioral group treatment for disruptive adolescents in a special school. Dissertation Abstracts International: B, 63(1–B), 528.

    Google Scholar 

  • Heflinger, C. A., Bickman, L., Northrup, D., & Sonnichsen, S. (1997). A theory-driven intervention and evaluation to explore family caregiver empowerment. Journal of Emotional and Behavioral Disorders, 5, 184–191.

    Google Scholar 

  • Henggeler, S. W., Melton, G. B., Brondino, M. J., Scherer, D. G., & Hanley, J. H. (1997). Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65, 821–833.

    Google Scholar 

  • Hoagwood, K., Burns, B., Kiser, L., Ringeisen, H., & Schoenwald, S. K. (2001). Evidence-based practice in child and adolescent mental health services. Psychiatric Services, 52, 1179–1189.

    Google Scholar 

  • Hoath, F. E., & Sanders, M. R. (2002). A feasibility study of enhanced group triple P—Positive Parenting Program for parents of children with attention-deficit(hyperactivity disorder. Behaviour Change, 19, 191–206.

    Google Scholar 

  • Huey, S. J., Jr., Henggeler, S. W., Rowland, M. D., Halliday-Boykins, C. A., Cunningham, P. B., Pickrel, S. G., et al. (2004) Multisystemic therapy effects on attempted suicide by youths presenting psychiatric emergencies. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 83–90

    Google Scholar 

  • Jensen, P. S., Hoagwood, K., & Trickett, E. (1999). Ivory Towers or Earthen Trenches? Community collaborations to foster real world research. Applied Developmental Science, 3, 306–212.

    Google Scholar 

  • Jocelyn, L. J., Casiro, O. G., Beattie, D., Bow, J., & Kneisz, J. (1998). Treatment of children with autism: A randomized controlled trial to evaluate a caregiver-based intervention program in community day-care centers. Journal of Developmental and Behavioral Pediatrics, 19, 326–334.

    Google Scholar 

  • Kaminer, Y., Blitz, C., Burleson, J. A., Kadden, R. M., & Rounsaville, B. J. (1998). Measuring treatment process in cognitive-behavioral and interactional group therapies for adolescent substance abusers. Journal of Nervous and Mental Diseases, 186, 407–413.

    Google Scholar 

  • Kaminer, Y., Burleson, J. A., Blitz, C., Sussman, J., & Rounsaville, B. J. (1998). Psychotherapies for adolescent substance abusers: A pilot study. Journal of Nervous and Mental Diseases, 186, 684–690.

    Google Scholar 

  • Kazdin, A. E. (2000). Developing a research agenda for child and adolescent psychotherapy. Archives of General Psychiatry, 57, 829–837.

    Google Scholar 

  • Kazdin, A. E., & Bass, D. (1989). Power to test differences between alternative treatments in comparative psychotherapy outcome research. Journal of Consulting and Clinical Psychology, 57, 138–147.

    Google Scholar 

  • Kazdin, A. E., & Nock, M. K. (2003). Delineating mechanisms of change in child and adolescent therapy: Methodological issues and research recommendations. Journal of Child Psychology and Psychiatry and Allied Disciplines, 44(8), 1116–1129.

    Google Scholar 

  • Kazdin, A. E., & Whitley, M. K. (2003). Treatment of parental stress to enhance therapeutic change among children referred for aggressive and antisocial behavior. Journal of Consulting and Clinical Psychology, 71, 504–515.

    Google Scholar 

  • King, N. J., Tonge, B. J., Mullen, P., Myerson, N., Heyne, D., Rollings, S., et al. (2000). Treating sexually abused children with posttraumatic-stress symptoms. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 1347–1355.

    Google Scholar 

  • Last, C. G., Hansen, C., & Franco, N. (1998). Cognitive-behavioral treatment of school phobia. Journal of the American Academy of Child and Adolescent Psychiatry, 37, 404–411.

    Google Scholar 

  • Liddle, H. A., Dakof, G. A., Parker, K., Diamond, G. S., Barrett, K., & Tejeda, M. (2001). Multidimiensional family therapy for adolescent drug abuse: Results of a randomized clinical trial. American Journal of Drug and Alcohol Abuse, 27, 651–688.

    Google Scholar 

  • McArdle, P., Moseley, D., Quibell, T., Johnson, R., Allen, A., Hammal, D., et al. (2002). School-based indicated prevention: a randomized trial of group therapy. Journal of Child Psychology and Psychiatry and Allied Disciplines, 43, 705–712.

    Google Scholar 

  • Mendlowitz, S. L., Manassis, K., Bradley, S., Scapillato, D., Miezitis, S., & Shaw, B. F. (1999). Cognitive behavioral group treatments in childhood anxiety disorders: The role of parental involvement. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1223–1229.

    Google Scholar 

  • Merkel, K. M. (2000). Effect of treatment of distress symptoms in child victims of violent crime: A comparison of supportive therapy and cognitive-behavioral therapy. Dissertation Abstracts International: B, 60(9-B), 4897.

    Google Scholar 

  • Montgomery, P., Stores, G., & Wiggs, L. (2004). The relative efficacy of two brief treatments for sleep problems in young learning disabled (mentally retarded) children: A randomised controlled trial. Archives of Disease in Childhood, 89, 125–130.

    Google Scholar 

  • Muris, P., Meesters, C., & van Melick, M. (2002). Treatment of childhood anxiety disorders; A preliminary comparison between cognitive-behavioral group therapy and a psychological placebo intervention. Journal of Behavior Therapy and Experimental Psychiatry, 33, 143–158.

    Google Scholar 

  • Murray, L., Cooper, P. J., Wilson, A., & Romaniuk, H. (2003). Controlled trial of the short- and long-term effect of psychological treatment of post-partum depression. 2. Impact on the mother–child relationship and child outcome. British Journal of Psychiatry, 182, 420–427.

    Google Scholar 

  • Nicholson, J. M., & Sanders, M. R. (1999). Randomized controlled trial of behavioral family intervention for the treatment of child behavior problems in stepfamilies. Journal of Divorce and Remarriage, 30, 1–23.

    Google Scholar 

  • Ost, L. G., Svensson, L., Hellstrom, K., & Lindwall, R. (2001). One-session treatment of specific phobias in youths: A randomized clinical trial. Journal of Consulting and Clinical Psychology, 69, 814–824.

    Google Scholar 

  • Robin, A. R., Siegel, P. T., Moye, A. W., Gilroy, M., Dennis, A. B., & Sikand, A. (1999). A controlled comparison of family versus individual therapy for adolescents with anorexia nervosa. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1482–1489.

    Google Scholar 

  • Rossello, J., & Bernal, G. (1999). The efficacy of cognitive, behavioral, and interpersonal treatments for depression in Puerto Rican adolescents. Journal of Consulting and Clinical Psychology, 67, 734–745.

    Google Scholar 

  • Scott, S., Spender, Q., Doolan, M., Jacobs, B., & Aspland, H. (2001). Multicentre controlled trial of parenting groups for childhood antisocial behaviour in clinical practice. British Medical Journal, 323, 194.

    Google Scholar 

  • Silverman, W. K., Kurtines, W. M., Ginsburg, G. S., Weems, C. F., Rabian, B., & Sarafini, L. T. (1999). Contingency management, self-control, and education support in the treatment of childhood phobic disorders: A randomized clinical trial. Journal of Consulting and Clinical Psychology, 67, 675–687.

    Google Scholar 

  • Snyder, K. V., Kymissis, P., & Kessler, K. (1999). Anger management for adolescents. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1409–1416.

    Google Scholar 

  • Sonuga-Barke, E. J. S., Daley, D., Thompson, M., Laver-Bradbury, C., & Weeks, A. (2001). Parent-based therapies for preschool attention-deficity(hyperactivity disorder: A randomized, controlled trial with a community sample. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 402–408.

    Google Scholar 

  • Spence, S. H., Donovan, C., & Brechman-Toussaint, M. (2000). The treatment of childhood social phobia: The effectiveness of a social-skills-training-based, cognitive-behavioral intervention, with and without parental involvement. Journal of Child Psychology and Psychiatry and Allied Disciplines, 41, 713–726.

    Google Scholar 

  • Spoth, R. L., Redmond, C., & Shin, C. (2001). Randomized trial of brief family interventions for general populations: Adolescent substance-use outcomes four years following baseline. Journal of Consulting and Clinical Psychology, 69, 662– 642.

    Google Scholar 

  • Storr, C. L., Ialongo, N. S., Kellam, S. G., & Anthony, J. C. (2002). A randomized controlled trial of two primary intervention strategies to prevent early onset tobacco smoking. Drug and Alcohol Dependence, 66, 51–60.

    Google Scholar 

  • Taylor, T. K., Schmidt, F., Pepler, D., & Hodgins, C. (1998). A comparison of eclectic treatment with Webster-Stratton’s parents and children series in a children’s mental health center: A randomized controlled trial. Behavior Therapy, 29, 221–240.

    Google Scholar 

  • Verduyn, C., Barrowclough, C., Roberts, J., Tarrier, N., & Harrington, R. (2003). Maternal depression and child behaviour problems: Randomised placebo-controlled trial of a cognitive-behavioural group intervention. British Journal of Psychiatry, 183, 342–348.

    Google Scholar 

  • Wayland, L. A. L. (2003). Treatment of children with problem behaviours: The efficacy of conjoint behavioural consultation versus videotape therapy and the impact on parent–teacher collaboration. Dissertation Abstracts International: Section B: The Sciences and Engineering, 64(1-B), 404.

    Google Scholar 

  • Weersing, V. R., & Weisz, J. R. (2002). Mechanisms of action of youth psychotherapy. Journal of Child Psychology & Psychiatry and Allied Disciplines, 43, 3–29.

    Google Scholar 

  • Weisz, J. R., & Jensen, P. S. (1999). Efficacy and effectiveness of psychotherapy and pharmacotherapy with children and adolescents. Mental Health Services Research, 1, 125–158.

    Google Scholar 

  • Weisz, J. R., Weiss, B., & Donenberg, G. R. (1992). The lab versus the clinic: Effects of child and adolescent psychotherapy. American Psychologist, 47, 1578–1585.

    Google Scholar 

  • Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2004). The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in clinical trials. Psychological Bulletin, 130, 631–663.

    Google Scholar 

  • Wiggs, L., & Stores, G. (1999). Behavioural treatment for sleep problems in children with severe learning disabilities and challenging daytime behaviour: Effect on daytime behaviour. Journal of Child Psychology and Psychiatry (Allied Disciplines, 40, 627–635.

    Google Scholar 

  • Wolchik, S. A., Sandler, I. N., Millsap, R. E., Plummer, B. A., Greene, S. M., Anderson, E. R., et al. (2002). Six-year follow-up of preventive interventions for children of divorce: a randomized controlled trial. JAMA, 288, 1874–1881.

    Google Scholar 

  • Wood, A., Harrington, R., & Moore, A. (1996). Controlled trial of a brief cognitive-behavioural intervention in adolescent patients with depressive disorders. Journal of Child Psychology and Psychiatry and Allied Disciplines, 37, 737–746.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Center for the Advancement of Children’s Mental Health at Columbia University/NYSPI, 1051 Riverside Drive, 10032, New York, New York

    Peter S. Jensen MD, Robin Weersing, Kimberly Eaton Hoagwood & Eliot Goldman

  2. Center for the Advancement of Children’s Mental Health at Columbia University/NYSPI, 1051 Riverside Drive, New York, New York

    Peter S. Jensen MD

Authors
  1. Peter S. Jensen MD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Robin Weersing
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kimberly Eaton Hoagwood
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eliot Goldman
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peter S. Jensen MD.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Jensen, P.S., Weersing, R., Hoagwood, K.E. et al. What Is the Evidence for Evidence-Based Treatments? A Hard Look at Our Soft Underbelly. Ment Health Serv Res 7, 53–74 (2005). https://doi.org/10.1007/s11020-005-1965-3

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11020-005-1965-3

Key words

Search

Navigation