Journal of Child and Family Studies

, Volume 18, Issue 6, pp 690–701 | Cite as

Identifying Youth at Risk for Treatment Failure in Outpatient Community Mental Health Services

  • Jared S. WarrenEmail author
  • Philip L. Nelson
  • Gary M. Burlingame
Original Paper


We developed predicted change trajectories and a warning system designed to identify psychotherapy cases at risk for treatment failure as observed in archival Youth Outcome Questionnaire data (parent/guardian-report) from 363 children and adolescents (ages 4–17) served in an outpatient community mental health system. We used multilevel modeling procedures to develop models of predicted change based on demographic information. Controlling for the effects of age on intercept, no other variables were significant in the model. The warning system we created from half of the sample (n = 181) correctly identified 71% of treatment failures in the other half of the sample (n = 182), defined as cases whose symptoms were significantly higher at the end of treatment compared to symptoms at intake. As over half of youth cases in this usual care setting did not demonstrate reliable improvement in symptoms, these results further emphasize the value of patient-focused research in monitoring patient progress and prompting changes in the treatment approach if suitable progress is not observed.


Treatment failure Change trajectories Usual care Patient-focused research Child psychotherapy 


  1. Achenbach, T. M. (1991). Manual for the child behavioral checklist/4–18 and 1991 profile. Burlington, VT: University of Vermont, Department of Psychiatry.Google Scholar
  2. Andrade, A. R., Lambert, E. W., & Bickman, L. (2000). Dose effect in child psychotherapy: Outcomes associated with negligible treatment. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 161–168. doi: 10.1097/00004583-200002000-00014.CrossRefPubMedGoogle Scholar
  3. APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. The American Psychologist, 61, 271–285. doi: 10.1037/0003-066X.61.4.271.CrossRefGoogle Scholar
  4. Baldwin, S. A., Berkeljon, A., Atkins, D. C., Olsen, J. A. & Nielsen, S. L. (in press). Rates of change in naturalistic psychotherapy: Contrasting dose-effect and good-enough level models of change. Journal of Consulting and Clinical Psychology.Google Scholar
  5. Bickman, L. (1996). A continuum of care: More is not always better. The American Psychologist, 51, 689–701. doi: 10.1037/0003-066X.51.7.689.CrossRefPubMedGoogle Scholar
  6. Bickman, L., Andrade, A. R., & Lambert, E. W. (2002). Dose response in child and adolescent mental health services. Mental Health Services Research, 4, 57–70. doi: 10.1023/A:1015210332175.CrossRefPubMedGoogle Scholar
  7. Bishop, M. J., Bybee, T. S., Lambert, M. J., Burlingame, G. M., Wells, M. G., & Poppleton, L. E. (2005). Accuracy of a rationally derived method for identifying treatment failure in children and adolescents. Journal of Child and Family Studies, 14, 207–222. doi: 10.1007/s10826-005-5049-1.CrossRefGoogle Scholar
  8. Breslin, F., Sobell, L. C., Buchan, G., & Cunningham, J. (1997). Toward a stepped-care approach to treating problem drinkers: The predictive validity of within-treatment variables and therapist prognostic ratings. Addiction (Abingdon, England), 92, 1479–1489. doi: 10.1111/j.1360-0443.1997.tb02869.x.Google Scholar
  9. Brown, G. S., Lambert, M. J., Jones, E. R., & Minami, T. (2005). Identifying highly effective psychotherapists in a managed care environment. The American Journal of Managed Care, 11, 513–520.PubMedGoogle Scholar
  10. Burlingame, G. M., Cox, J. C., Wells, M. G., Lambert, M. J., Latkowski, M., & Ferre, R. (2005). The administration and scoring manual of the Youth Outcome Questionnaire. Salt Lake City, UT: American Professional Credentialing Services.Google Scholar
  11. Burlingame, G. M., Mosier, J. I., Wells, M. G., Atkin, Q. G., Lambert, M. J., Whoolery, M., et al. (2001). Tracking the influence of mental health treatment: The development of the Youth Outcome Questionnaire. Clinical Psychology & Psychotherapy, 8, 361–379. doi: 10.1002/cpp.315.CrossRefGoogle Scholar
  12. Burlingame, G. M., Wells, A., Lambert, M. J., & Cox, J. (2004). Youth Outcome Questionnaire: Updated psychometric properties. In M. E. Maruish (Ed.), The use of psychological testing for treatment planning and outcome assessment (3rd ed., Vol. 4, pp. 235–274). Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
  13. Bybee, T. S., Lambert, M. J., & Eggett, D. (2007). Curves of expected recovery and their predictive validity for identifying treatment failure. Dutch Journal of Psychotherapy, 33, 419–434.Google Scholar
  14. Cannon, J. A. N., Warren, J. S., Nelson, P. L., & Burlingame, G. M. (2009). Identifying youth at risk for psychotherapy treatment failure: Expected change trajectories for the Youth Outcome Questionnaire self-report. Manuscript submitted for publication.Google Scholar
  15. Conners, C. K., Sitarenios, G., Parker, J. D., & Epstein, J. N. (1998). The revised Conners’ parent rating scale (CPRS–R): Factor structure, reliability, and criterion validity. Journal of Abnormal Child Psychology, 26, 257–268. doi: 10.1023/A:1022602400621.CrossRefPubMedGoogle Scholar
  16. Finch, A. E., Lambert, M. J., & Schaalje, B. G. (2001). Psychotherapy quality control: The statistical generation of expected recovery curves for integration into an early warning system. Clinical Psychology & Psychotherapy, 8, 231–242. doi: 10.1002/cpp.286.CrossRefGoogle Scholar
  17. Grove, W. M., & Meehl, P. E. (1996). Comparative efficiency of informal (subjective, impressionistic) and formal (mechanical, algorithmic) prediction procedures: The clinical-statistical controversy. Psychology, Public Policy, and Law, 2, 293–323. doi: 10.1037/1076-8971.2.2.293.CrossRefGoogle Scholar
  18. Hannan, C., Lambert, M. J., Harmon, C., Nielsen, S. L., Smart, D. W., Shimokawa, K., et al. (2005). A lab test and algorithms for identifying cases at risk for treatment failure. Journal of Clinical Psychology, 61, 155–163. doi: 10.1002/jclp.20108.CrossRefPubMedGoogle Scholar
  19. Harmon, S. C., Lambert, M. J., Smart, D. M., Hawkins, E., Nielsen, S. L., Slade, K., et al. (2007). Enhancing outcome for potential treatment failures: Therapist—client feedback and clinical support tools. Psychotherapy Research, 17, 379–392. doi: 10.1080/10503300600702331.CrossRefGoogle Scholar
  20. Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K., & Tuttle, K. (2004). The therapeutic effects of providing client progress information to patients and therapists. Psychotherapy Research, 10, 308–327. doi: 10.1093/ptr/kph027.CrossRefGoogle Scholar
  21. Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z., & Lutz, W. (1996). Evaluation of psychotherapy: Efficacy, effectiveness, and patient progress. The American Psychologist, 51, 1059–1064. doi: 10.1037/0003-066X.51.10.1059.CrossRefPubMedGoogle Scholar
  22. Institute of Medicine. (2006). Improving the quality of health care for mental and substance-use conditions: Quality chasm series. Washington, DC: National Academies Press.Google Scholar
  23. Jacobson, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59, 12–19. doi: 10.1037/0022-006X.59.1.12.CrossRefPubMedGoogle Scholar
  24. Jensen, A. L., & Weisz, J. R. (2002). Assessing match and mismatch between practitioner-generated and standardized interview-generated diagnoses for clinic-referred children and adolescents. Journal of Consulting and Clinical Psychology, 70, 158–168. doi: 10.1037/0022-006X.70.1.158.CrossRefPubMedGoogle Scholar
  25. Kazdin, A. E. (1996). Dropping out of child therapy: Issues for research and implications for practice. Clinical Child Psychology and Psychiatry, 1, 133–156. doi: 10.1177/1359104596011012.CrossRefGoogle Scholar
  26. Kazdin, A. E. (2005). Evidence-based assessment for children and adolescents: Issues in measurement development and clinical application. Journal of Clinical Child and Adolescent Psychology, 34, 548–558. doi: 10.1207/s15374424jccp3403_10.CrossRefPubMedGoogle Scholar
  27. Lambert, M. J., & Bergin, A. E. (1994). The effectiveness of psychotherapy. In A. E. Bergin & S. L. Garfield (Eds.), Handbook of psychotherapy and behavior change (4th ed., pp. 143–189). New York: John Wiley and Sons.Google Scholar
  28. Lambert, M. J., Hansen, N. B., & Finch, A. E. (2001). Patient-focused research: Using patient outcome data to enhance treatment effects. Journal of Consulting and Clinical Psychology, 69, 159–172. doi: 10.1037/0022-006X.69.2.159.CrossRefPubMedGoogle Scholar
  29. Lambert, M. J., Morton, J. J., Hatfield, D., Harmon, C., Hamilton, S., Reid, R. C., et al. (2004). Administration and scoring manual for the outcome questionnaire -45. Orem, UT: American Professional Credentialing Services.Google Scholar
  30. Lambert, M. J., Whipple, J. L., Bishop, M. J., Vermeersch, D. A., Gray, G. V., & Finch, A. E. (2002a). Comparison of empirically-derived and rationally-derived methods for identifying patients at risk for treatment failure. Clinical Psychology & Psychotherapy, 9, 149–164. doi: 10.1002/cpp.333.CrossRefGoogle Scholar
  31. Lambert, M. J., Whipple, J. L., Vermeersch, D. A., Smart, D. W., Hawkins, E. J., Nielsen, S. L., et al. (2002b). Enhancing psychotherapy outcomes via providing feedback on client progress: A replication. Clinical Psychology & Psychotherapy, 9, 91–103. doi: 10.1002/cpp.324.CrossRefGoogle Scholar
  32. Lilienfeld, S. O. (2007). Psychological treatments that cause harm. Perspectives on Psychological Science, 2, 53–70. doi: 10.1111/j.1745-6916.2007.00029.x.CrossRefGoogle Scholar
  33. Lueger, R. J., Howard, K. I., Martinovich, Z., Lutz, W., Anderson, E. E., & Grissom, G. (2001). Assessing treatment progress of individual patients using expected treatment response models. Journal of Consulting and Clinical Psychology, 69, 150–158. doi: 10.1037/0022-006X.69.2.150.CrossRefPubMedGoogle Scholar
  34. Lutz, W., Martinovich, Z., & Howard, K. I. (1999). Patient profiling: An application of random coefficient regression models to depicting the response of a patient to outpatient psychotherapy. Journal of Consulting and Clinical Psychology, 67, 571–577. doi: 10.1037/0022-006X.67.4.571.CrossRefPubMedGoogle Scholar
  35. Mohr, D. C. (1995). Negative outcome in psychotherapy: A critical review. Clinical Psychology: Science and Practice, 2, 1–27.CrossRefGoogle Scholar
  36. National Advisory Mental HealthCouncil. (2001). Blueprint for change: Research on child and adolescent mental health. A report by the national advisory mental health council’s workgroup on child and adolescent mental health intervention development and deployment. Bethesda. MD: National Institutes of Health/National Institute of Mental Health.Google Scholar
  37. Ringel, J. S., & Sturm, R. (2001). National estimates of mental health utilization and expenditures for children in 1998. J. Behav. Health Serv. Res., 28, 319–333. doi: 10.1007/BF02287247.CrossRefPubMedGoogle Scholar
  38. Salzer, M. S., Bickman, L., & Lambert, E. W. (1999). Dose-effect relationship in children’s psychotherapy services. Journal of Consulting and Clinical Psychology, 67, 228–238. doi: 10.1037/0022-006X.67.2.228.CrossRefPubMedGoogle Scholar
  39. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York: Oxford.Google Scholar
  40. Spielmans, G. I., Masters, K. S., & Lambert, M. J. (2006). A comparison of rational versus empirical methods in the prediction of psychotherapy outcome. Clinical Psychology & Psychotherapy, 13, 202–214. doi: 10.1002/cpp.491.CrossRefGoogle Scholar
  41. Weiss, B., Catron, T., Harris, V., & Phung, T. M. (1999). The effectiveness of traditional child psychotherapy. Journal of Consulting and Clinical Psychology, 67, 82–94. doi: 10.1037/0022-006X.67.1.82.CrossRefPubMedGoogle Scholar
  42. Weisz, J. R. (2004). Psychotherapy for children and adolescents: Evidence-based treatments and case examples. Cambridge: Cambridge University Press.Google Scholar
  43. Weisz, J. R., Donenberg, G. R., Han, S. S., & Weiss, B. (1995). Bridging the gap between lab and clinic in child and adolescent psychotherapy. Journal of Consulting and Clinical Psychology, 63, 688–701. doi: 10.1037/0022-006X.63.5.688.CrossRefPubMedGoogle Scholar
  44. Whipple, J. L., Lambert, M. J., Vermeersch, D. A., Smart, D. W., Nielsen, S. L., & Hawkins, E. J. (2003). Improving the effects of psychotherapy: The use of early identification of treatment failure and problem solving strategies in routine practice. Journal of Counseling Psychology, 50(1), 59–68. doi: 10.1037/0022-0167.50.1.59.CrossRefGoogle Scholar
  45. Wierzbicki, M., & Pekarik, G. (1993). A meta-analysis of psychotherapy dropout. Professional Psychology, Research and Practice, 24, 190–195. doi: 10.1037/0735-7028.24.2.190.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Jared S. Warren
    • 1
    Email author
  • Philip L. Nelson
    • 2
  • Gary M. Burlingame
    • 1
  1. 1.Department of Psychology and Clinical PsychologyBrigham Young UniversityProvoUSA
  2. 2.Department of Counseling Psychology & Special EducationBrigham Young UniversityProvoUSA

Personalised recommendations