Mental Health Services Research

, Volume 7, Issue 1, pp 53–74

What Is the Evidence for Evidence-Based Treatments? A Hard Look at Our Soft Underbelly

  • Peter S. Jensen
  • Robin Weersing
  • Kimberly Eaton Hoagwood
  • Eliot Goldman
Article

Abstract

In the rising quest for evidence-based interventions, recent research often does not give adequate attention to “nonspecific therapeutic factors,” including the effects of attention, positive regard, and therapeutic alliance, as well as the effects of treatment dose, intensity and actual processes mediating therapeutic change. To determine the extent to which recent clinical trial designs fully this problem, the authors conducted a systematic review of PsychLit/Medline of all controlled child psychotherapy treatment studies from 1995 to 2004. A total of 52 studies were identified that met review criteria: two or more therapy conditions and random assignment of participants to intervention groups. Of the 52 studies, one group (n = 29) compared a presumably active treatment with 1 or more similarly intensive treatments (often an attention control group) presumably not containing the active therapeutic ingredients. Of these, 14 studies found evidence of consistent differences between the two groups, whereas 15 did not. An additional group of studies (n = 27) compared therapy groups with different levels of intensity and “dose” of the putatively active treatment; 13 of these found evidence of the effects of different levels of treatment dose/intensity on outcomes and 14 did not. Four studies met criteria for inclusion in both groups. Across both groups of studies, when positive effects were found, few studies systematically explored whether the presumed active therapeutic ingredients actually accounted for the degree of change, nor did they often address plausible alternative explanations, such as nonspecific therapeutic factors of positive expectancies, therapeutic alliance, or attention. Findings suggest that many child psychotherapy treatment studies have not inadequately controlled for nonspecific factors such as attention and treatment intensity and have failed to assess specific mediators of change. Specific recommendations for future studies are offered, specifically:

  1. 1.

    Initial specification in study design how investigators will test if the intervention is efficacious over and above the effects of positive expectancies, positive regard, or attention;

     
  2. 2.

    Planned (rather than post hoc) analyses to explore whether and how a given treatment’s specific ingredients or the overall intensity of its “dose” are related to treatment outcomes;

     
  3. 3.

    More studies conducting head-to-head tests of different types but equally credible forms of treatment, with planned analyses testing different mechanisms of change

     
  4. 4.

    When positive effects of a treatment vs. a control are found, systematic elimination of specific, competing hypotheses of reasons for treatment efficacy (e.g., attention, therapeutic alliance, face validity of treatment and client’s treatment expectations, changes in self-efficacy), and

     
  5. 5.

    When negative effects are reported, appropriate analyses to address alternative explanations (lack of power, floor or ceiling effects, mediator analyses to address possible fidelity or adherence problems, therapist effects, absence of main effects but possible subgroup/moderator effects, etc.).

     

Key words

attention controls efficacy psychotherapy controlled clinical trials 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abikoff, H., Hechtman, L., Klein, R. G., Weiss, G., Fleiss, K., Etcovitch, J., et al. (2004). Symptomatic improvement in children with ADHD treated with long-term methylphenidate and multimodal psychosocial treatment. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 802–811.Google Scholar
  2. Barrett, P. M. (1998). Evaluation of cognitive-behavioral group treatments for childhood anxiety disorders. Journal of Clinical Child Psychology, 27, 459–468.Google Scholar
  3. Barrett, P., Healy-Farrell, L., & March, J. S. (2004). Cognitive-behavioral family treatment of childhood obsessive-compulsive disorder: A controlled trial. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 46–62.Google Scholar
  4. Barrett, P. M., Dadds, M. R., & Rapee, R. M. (1996). Family treatment of childhood anxiety: A controlled trial. Journal of Consulting and Clinical Psychology, 64, 333–342.Google Scholar
  5. Baskin, T. W., Tierney, S. C., Minami, T., & Wampold, B. E. (2003). Establishing specificity in psychotherapy: A meta-analysis of structural equivalence of placebo controls. Journal of Consulting and Clinical Psychology, 71, 973–979.Google Scholar
  6. Beidel, D. C., Turner, S. M., & Morris, T. L. (2000). Behavioral treatment of childhood social phobia. Journal of Consulting and Clinical Psychology, 68, 1072–1080.Google Scholar
  7. Bickman, L., Heflinger, C. A., Northrup, D., Sonnichsen, S., & Schilling, S. (1998). Long-term outcomes of family caregiver empowerment. Journal of Child and Family Studies, 7, 269–282.Google Scholar
  8. Birmaher, B., Brent, D. A., Kolko, D., Baugher, M., Bridge, J., Holder, D., et al. (2000), Clinical outcome after short-term psychotherapy for adolescents with major depressive disorder. Archives of General Psychiatry, 57, 29–36.Google Scholar
  9. Brent, D. A., Holder, D., Kolko, D., Birmaher, B., Baugher, M., Roth, C., et al. (1997). A clinical psychotherapy trial for adolescent depression comparing cognitive, family, and supportive therapy. Archives of General Psychiatry, 54, 877–885.Google Scholar
  10. Chadwick, O., Momcilovic, N., Rossiter, R., Stumbles, E., & Taylor, E. (2001). A randomized trial of brief individual versus group parent training for behaviour problems in children with severe learning disabilities. Behavioural and Cognitive Psychotherapy, 29, 151–167.Google Scholar
  11. Chamberlain, P., & Reid, J. B. (1998). Comparison of two community alternatives to incarceration for chronic juvenile offenders. Journal of Consulting and Clinical Psychology, 66, 624–633.Google Scholar
  12. Chambless, D., & Hollon, S. D. (1998). Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7–18.Google Scholar
  13. Clarke, G. N., Hornbrook, M., Lynch, F., & Polen, M. (2002). Group cognitive-behavioral treatment for depressed adolescent offspring of depressed parents in a health maintenance organization. Journal of the American Academy of Child and Adolescent Psychiatry, 41, 305–313.Google Scholar
  14. Clarke, G. N., Rohde, P., Lewinsohn, P. M., Hops, H., & Seeley, J. R. (1999). Cognitive-behavioral treatment of adolescent depression: Efficacy of acute group treatment and booster sessions. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 272–279.Google Scholar
  15. Cobham, V. E., Dadds, M. R., & Spence, S. H. (1998). The role of parental anxiety in the treatment of childhood anxiety. Journal of Consulting and Clinical Psychology, 66, 893–905.Google Scholar
  16. Cohen, J. (1988). Statistical power analyses for the behavioral sciences (2nd edn.). Hillsdale, NJ: Lawrence Erlbaum.Google Scholar
  17. Cohen, J. A., & Mannarino, A. P. (1996). A treatment outcome study for sexually abused preschool children. Journal of the American Academy of Child and Adolescent Psychiatry, 35, 42–50.Google Scholar
  18. Cohen, N. J., Muir, E., Parker, C. J., Brown, M., Lojkasek, M., Muir, R., et al. (1999). Watch, wait and wonder: Testing the effectiveness of a new approach to mother-infant psychotherapy. Infant Mental Health Journal, 20, 429–451.Google Scholar
  19. Connolly, L., Sharry, J., & Fitzpatrick, C. (2001). Evaluation of a group treatment programme for parents of children with behavioural disorders. Child and Adolescent Mental Health, 6, 159–165.Google Scholar
  20. Dadds, M. R., Spence, S. H., Holland, D. E., Barrett, P. M., & Laurens, K. R. (1997). Prevention and early intervention for anxiety disorders: A controlled trial. Journal of Consulting and Clinical Psychology, 65, 627–635.Google Scholar
  21. Dishion, T. J., & Andrews, D. W. (1995). Preventing escalation in problem behaviors with high-risk young adolescents: immediate and 1-year outcomes. Journal of Consulting and Clinical Psychology, 63, 538–548Google Scholar
  22. dos Santos, E. L. C., Marturano, E. M., de Almeida Motta, A. M., & Giurlana, A. G. (2003). Treating boys with low school achievement and behavior problems: Comparison of two kinds of intervention. Psychological Reports, 92, 105–116.Google Scholar
  23. Driskill, J. D. (2000). Structured child and parent groups with ADHD children: Evaluation of varying levels of parent involvement. Dissertation Abstracts International: B, 60(9–B), 4884.Google Scholar
  24. Eddy, J. M., & Chamberlain, P. (2000). Family management and deviant peer association as mediators of the impact of treatment condition on youth antisocial behavior. Journal of Consulting and Clinical Psychology, 68, 857–863.Google Scholar
  25. Eddy, J. M., Whaley, R. B., & Chamberlain, P. (2004). The prevention of violent behavior by chronic and serious male juvenile offenders: A 2-year follow-up of a randomized clinical trial. Journal of Emotional and Behavioral Disorders, 12, 2–8.Google Scholar
  26. Eisler, I., Dare, C., Hodes, M., Russell, G., Dodge, E., & Le Grange, D. (2000). Family therapy for adolescent anorexia nervosa: The results of a controlled comparison of two family interventions. Journal of Child Psychology and Psychiatry and Allied Disciplines, 41, 727–736.Google Scholar
  27. Flannery-Schroeder, E. C., & Kendall, P. C. (2000). Group and individual cognitive-behavioral treatments for youth with anxiety disorders: A randomized clinical trial. Cognitive Therapy and Research, 24, 251–278.Google Scholar
  28. Forgatch, M. S., & DeGarmo, D. S. (1999). Parenting through change: An effective prevention program for single mothers. Journal of Consulting and Clinical Psychology, 67, 711–724.Google Scholar
  29. Hebert, D. A. (2002). Evaluating cognitive-behavioral group treatment for disruptive adolescents in a special school. Dissertation Abstracts International: B, 63(1–B), 528.Google Scholar
  30. Heflinger, C. A., Bickman, L., Northrup, D., & Sonnichsen, S. (1997). A theory-driven intervention and evaluation to explore family caregiver empowerment. Journal of Emotional and Behavioral Disorders, 5, 184–191.Google Scholar
  31. Henggeler, S. W., Melton, G. B., Brondino, M. J., Scherer, D. G., & Hanley, J. H. (1997). Multisystemic therapy with violent and chronic juvenile offenders and their families: The role of treatment fidelity in successful dissemination. Journal of Consulting and Clinical Psychology, 65, 821–833.Google Scholar
  32. Hoagwood, K., Burns, B., Kiser, L., Ringeisen, H., & Schoenwald, S. K. (2001). Evidence-based practice in child and adolescent mental health services. Psychiatric Services, 52, 1179–1189.Google Scholar
  33. Hoath, F. E., & Sanders, M. R. (2002). A feasibility study of enhanced group triple P—Positive Parenting Program for parents of children with attention-deficit(hyperactivity disorder. Behaviour Change, 19, 191–206.Google Scholar
  34. Huey, S. J., Jr., Henggeler, S. W., Rowland, M. D., Halliday-Boykins, C. A., Cunningham, P. B., Pickrel, S. G., et al. (2004) Multisystemic therapy effects on attempted suicide by youths presenting psychiatric emergencies. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 83–90Google Scholar
  35. Jensen, P. S., Hoagwood, K., & Trickett, E. (1999). Ivory Towers or Earthen Trenches? Community collaborations to foster real world research. Applied Developmental Science, 3, 306–212.Google Scholar
  36. Jocelyn, L. J., Casiro, O. G., Beattie, D., Bow, J., & Kneisz, J. (1998). Treatment of children with autism: A randomized controlled trial to evaluate a caregiver-based intervention program in community day-care centers. Journal of Developmental and Behavioral Pediatrics, 19, 326–334.Google Scholar
  37. Kaminer, Y., Blitz, C., Burleson, J. A., Kadden, R. M., & Rounsaville, B. J. (1998). Measuring treatment process in cognitive-behavioral and interactional group therapies for adolescent substance abusers. Journal of Nervous and Mental Diseases, 186, 407–413.Google Scholar
  38. Kaminer, Y., Burleson, J. A., Blitz, C., Sussman, J., & Rounsaville, B. J. (1998). Psychotherapies for adolescent substance abusers: A pilot study. Journal of Nervous and Mental Diseases, 186, 684–690.Google Scholar
  39. Kazdin, A. E. (2000). Developing a research agenda for child and adolescent psychotherapy. Archives of General Psychiatry, 57, 829–837.Google Scholar
  40. Kazdin, A. E., & Bass, D. (1989). Power to test differences between alternative treatments in comparative psychotherapy outcome research. Journal of Consulting and Clinical Psychology, 57, 138–147.Google Scholar
  41. Kazdin, A. E., & Nock, M. K. (2003). Delineating mechanisms of change in child and adolescent therapy: Methodological issues and research recommendations. Journal of Child Psychology and Psychiatry and Allied Disciplines, 44(8), 1116–1129.Google Scholar
  42. Kazdin, A. E., & Whitley, M. K. (2003). Treatment of parental stress to enhance therapeutic change among children referred for aggressive and antisocial behavior. Journal of Consulting and Clinical Psychology, 71, 504–515.Google Scholar
  43. King, N. J., Tonge, B. J., Mullen, P., Myerson, N., Heyne, D., Rollings, S., et al. (2000). Treating sexually abused children with posttraumatic-stress symptoms. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 1347–1355.Google Scholar
  44. Last, C. G., Hansen, C., & Franco, N. (1998). Cognitive-behavioral treatment of school phobia. Journal of the American Academy of Child and Adolescent Psychiatry, 37, 404–411.Google Scholar
  45. Liddle, H. A., Dakof, G. A., Parker, K., Diamond, G. S., Barrett, K., & Tejeda, M. (2001). Multidimiensional family therapy for adolescent drug abuse: Results of a randomized clinical trial. American Journal of Drug and Alcohol Abuse, 27, 651–688.Google Scholar
  46. McArdle, P., Moseley, D., Quibell, T., Johnson, R., Allen, A., Hammal, D., et al. (2002). School-based indicated prevention: a randomized trial of group therapy. Journal of Child Psychology and Psychiatry and Allied Disciplines, 43, 705–712.Google Scholar
  47. Mendlowitz, S. L., Manassis, K., Bradley, S., Scapillato, D., Miezitis, S., & Shaw, B. F. (1999). Cognitive behavioral group treatments in childhood anxiety disorders: The role of parental involvement. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1223–1229.Google Scholar
  48. Merkel, K. M. (2000). Effect of treatment of distress symptoms in child victims of violent crime: A comparison of supportive therapy and cognitive-behavioral therapy. Dissertation Abstracts International: B, 60(9-B), 4897.Google Scholar
  49. Montgomery, P., Stores, G., & Wiggs, L. (2004). The relative efficacy of two brief treatments for sleep problems in young learning disabled (mentally retarded) children: A randomised controlled trial. Archives of Disease in Childhood, 89, 125–130.Google Scholar
  50. Muris, P., Meesters, C., & van Melick, M. (2002). Treatment of childhood anxiety disorders; A preliminary comparison between cognitive-behavioral group therapy and a psychological placebo intervention. Journal of Behavior Therapy and Experimental Psychiatry, 33, 143–158.Google Scholar
  51. Murray, L., Cooper, P. J., Wilson, A., & Romaniuk, H. (2003). Controlled trial of the short- and long-term effect of psychological treatment of post-partum depression. 2. Impact on the mother–child relationship and child outcome. British Journal of Psychiatry, 182, 420–427.Google Scholar
  52. Nicholson, J. M., & Sanders, M. R. (1999). Randomized controlled trial of behavioral family intervention for the treatment of child behavior problems in stepfamilies. Journal of Divorce and Remarriage, 30, 1–23.Google Scholar
  53. Ost, L. G., Svensson, L., Hellstrom, K., & Lindwall, R. (2001). One-session treatment of specific phobias in youths: A randomized clinical trial. Journal of Consulting and Clinical Psychology, 69, 814–824.Google Scholar
  54. Robin, A. R., Siegel, P. T., Moye, A. W., Gilroy, M., Dennis, A. B., & Sikand, A. (1999). A controlled comparison of family versus individual therapy for adolescents with anorexia nervosa. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1482–1489.Google Scholar
  55. Rossello, J., & Bernal, G. (1999). The efficacy of cognitive, behavioral, and interpersonal treatments for depression in Puerto Rican adolescents. Journal of Consulting and Clinical Psychology, 67, 734–745.Google Scholar
  56. Scott, S., Spender, Q., Doolan, M., Jacobs, B., & Aspland, H. (2001). Multicentre controlled trial of parenting groups for childhood antisocial behaviour in clinical practice. British Medical Journal, 323, 194.Google Scholar
  57. Silverman, W. K., Kurtines, W. M., Ginsburg, G. S., Weems, C. F., Rabian, B., & Sarafini, L. T. (1999). Contingency management, self-control, and education support in the treatment of childhood phobic disorders: A randomized clinical trial. Journal of Consulting and Clinical Psychology, 67, 675–687.Google Scholar
  58. Snyder, K. V., Kymissis, P., & Kessler, K. (1999). Anger management for adolescents. Journal of the American Academy of Child and Adolescent Psychiatry, 38, 1409–1416.Google Scholar
  59. Sonuga-Barke, E. J. S., Daley, D., Thompson, M., Laver-Bradbury, C., & Weeks, A. (2001). Parent-based therapies for preschool attention-deficity(hyperactivity disorder: A randomized, controlled trial with a community sample. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 402–408.Google Scholar
  60. Spence, S. H., Donovan, C., & Brechman-Toussaint, M. (2000). The treatment of childhood social phobia: The effectiveness of a social-skills-training-based, cognitive-behavioral intervention, with and without parental involvement. Journal of Child Psychology and Psychiatry and Allied Disciplines, 41, 713–726.Google Scholar
  61. Spoth, R. L., Redmond, C., & Shin, C. (2001). Randomized trial of brief family interventions for general populations: Adolescent substance-use outcomes four years following baseline. Journal of Consulting and Clinical Psychology, 69, 662– 642.Google Scholar
  62. Storr, C. L., Ialongo, N. S., Kellam, S. G., & Anthony, J. C. (2002). A randomized controlled trial of two primary intervention strategies to prevent early onset tobacco smoking. Drug and Alcohol Dependence, 66, 51–60.Google Scholar
  63. Taylor, T. K., Schmidt, F., Pepler, D., & Hodgins, C. (1998). A comparison of eclectic treatment with Webster-Stratton’s parents and children series in a children’s mental health center: A randomized controlled trial. Behavior Therapy, 29, 221–240.Google Scholar
  64. Verduyn, C., Barrowclough, C., Roberts, J., Tarrier, N., & Harrington, R. (2003). Maternal depression and child behaviour problems: Randomised placebo-controlled trial of a cognitive-behavioural group intervention. British Journal of Psychiatry, 183, 342–348.Google Scholar
  65. Wayland, L. A. L. (2003). Treatment of children with problem behaviours: The efficacy of conjoint behavioural consultation versus videotape therapy and the impact on parent–teacher collaboration. Dissertation Abstracts International: Section B: The Sciences and Engineering, 64(1-B), 404.Google Scholar
  66. Weersing, V. R., & Weisz, J. R. (2002). Mechanisms of action of youth psychotherapy. Journal of Child Psychology & Psychiatry and Allied Disciplines, 43, 3–29.Google Scholar
  67. Weisz, J. R., & Jensen, P. S. (1999). Efficacy and effectiveness of psychotherapy and pharmacotherapy with children and adolescents. Mental Health Services Research, 1, 125–158.Google Scholar
  68. Weisz, J. R., Weiss, B., & Donenberg, G. R. (1992). The lab versus the clinic: Effects of child and adolescent psychotherapy. American Psychologist, 47, 1578–1585.Google Scholar
  69. Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2004). The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in clinical trials. Psychological Bulletin, 130, 631–663.Google Scholar
  70. Wiggs, L., & Stores, G. (1999). Behavioural treatment for sleep problems in children with severe learning disabilities and challenging daytime behaviour: Effect on daytime behaviour. Journal of Child Psychology and Psychiatry (Allied Disciplines, 40, 627–635.Google Scholar
  71. Wolchik, S. A., Sandler, I. N., Millsap, R. E., Plummer, B. A., Greene, S. M., Anderson, E. R., et al. (2002). Six-year follow-up of preventive interventions for children of divorce: a randomized controlled trial. JAMA, 288, 1874–1881.Google Scholar
  72. Wood, A., Harrington, R., & Moore, A. (1996). Controlled trial of a brief cognitive-behavioural intervention in adolescent patients with depressive disorders. Journal of Child Psychology and Psychiatry and Allied Disciplines, 37, 737–746.Google Scholar

Copyright information

© Springer Science + Business Media, Inc. 2005

Authors and Affiliations

  • Peter S. Jensen
    • 1
    • 2
  • Robin Weersing
    • 1
  • Kimberly Eaton Hoagwood
    • 1
  • Eliot Goldman
    • 1
  1. 1.Center for the Advancement of Children’s Mental Health at Columbia University/NYSPINew YorkNew York
  2. 2.Center for the Advancement of Children’s Mental Health at Columbia University/NYSPINew YorkNew York

Personalised recommendations