Child & Youth Care Forum

, Volume 47, Issue 3, pp 403–419 | Cite as

A Pilot Study Examining Outcomes Associated with the Implementation of Progress Monitoring at a Substance Use Disorder Treatment Program for Adolescents

  • Keith C. RussellEmail author
  • Harold L. Gillis
  • Liam Law
  • Jeff Couillard
Original Paper



Progress monitoring (PM) is the periodic and reliable assessment of client progress to evaluate and inform psychotherapeutic treatment. PM’s use in a variety of treatment contexts show improved treatment outcome, dropout rate reduction, moderated treatment deterioration, and more efficient treatment delivery.


This study reports the preliminary results of a PM implementation initiative at a youth care program that focuses on substance use disorder treatment.


Participants were 70 adolescent clients (mean age 16.6 years, 56% male, 58% White, 30% Aboriginal or Mètis). A time series evaluation design guided the study, where data were collected at admission, periodic intervals during treatment, and at discharge. Descriptive analysis was performed on all demographic variables. Pairwise t tests and effect sizes were calculated to determine clinically significant change.


PM was successfully implemented in the youth care program and the outcomes associated with PM were statistically and clinically significant with large effect sizes reported. Changes in the development of mindfulness related-skills were positively associated with changes in outcome. Case studies illustrate how PM can identify clients who are not responding to treatment.


PM in adolescent substance use treatment programs is an important consideration for program providers. Implementation needs to involve staff, be flexible, empower clients, and be integrated into the culture of programs. The successful implementation of PM can help identify clients who are not responding to treatment and generate useful and reliable outcome data. Recommendations for how PM can be implemented in similar youth care programs are also discussed.


Progress monitoring Adventure therapy Youth Outcome Questionnaire Mindfulness practices 



This study was funded by a grant from Health Canada and Alberta Health Services (Grant# 005470). K.C. Russell and H.L. Gillis, received funding to conduct this evaluation from Health Canada and Alberta Health Services (Grant# 005470). L. Law and J. Couillard both work for ENVIROS, the Organization with which the research was coordinated and conducted.

Compliance with Ethical Standards

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the study and the study was approved by the Institutional Review Board of Georgia College and State University (#656206: Evaluation of Enviros Base Camp Program Co-Principal Investigator: Lee Gillis, Ph.D.).


  1. American Psychological Association. (2009). Criteria for the evaluation of quality improvement programs and the use of quality improvement data. American Psychologist, 64, 551–557. Scholar
  2. Base Camp. (2017). Retrieved November 20, 2017, from
  3. Bedard, R. M., Rosen, L. A., & Vacha-Haase, T. (2003). Wilderness therapy programs for juvenile delinquents: A meta-analysis. Journal of Therapeutic Wilderness Camping, 3, 7–13.Google Scholar
  4. Bettmann, J. E., Gillis, H. L., Speelman, E. A., Parry, K. J., & Case, J. M. (2016). A meta-analysis of wilderness therapy outcomes for private pay clients. Journal of Child and Family Studies, 25(9), 2659–2673.CrossRefGoogle Scholar
  5. Bickman, L., Kelley, S. D., Breda, C., Regina de Andrade, A., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62(12), 1423–1429. Scholar
  6. Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015). Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy Research: Journal of the Society for Psychotherapy Research. Scholar
  7. Bowen, D. J., & Neill, J. T. (2013). A meta-analysis of adventure therapy outcomes and moderators. The Open Psychology Journal, 6, 28–53.CrossRefGoogle Scholar
  8. Burlingame, G., Cox, J., Wells, G., Latkowski, M., Justice, D., Carter, C., et al. (2005). The administration and scoring manual of the Youth Outcome Questionnaire. Salt Lake City, Utah: OQ Measures.Google Scholar
  9. Carlier, I. V., Meuldijk, D., Van Vliet, I. M., Van Fenema, E., Van der Wee, N. J. A., & Zitman, F. G. (2012). ROM and feedback on physical or mental health status: Evidence and theory. Journal of Evaluation in Clinical Practice, 18, 104–110. Scholar
  10. Cason, D., & Gillis, H. L. (1994). A meta-analysis of outdoor adventure programming with adolescents. Journal of Experiential Education, 17, 40–47.CrossRefGoogle Scholar
  11. Duncan, B. L., & Shaw, G. B. (2012). The partners for change outcome management system (PCOMS): The heart and soul of change project. Canadian Psychology/Psychologie Canadienne, 53(2), 93–104. Scholar
  12. Fernandez, A. C., Wood, M. D., Stein, L. A. R., & Rossi, J. S. (2010). Measuring mindfulness and examining its relationship with alcohol use and negative consequences. Psychology of Addictive Behaviors : Journal of the Society of Psychologists in Addictive Behaviors, 24(4), 608–616.CrossRefGoogle Scholar
  13. Gass, M., Gillis, H. L., & Russell, K. C. (2012). Adventure therapy: Theory, practice and research. New York: Routledge Mental Health Publishing.Google Scholar
  14. Gillis, H. L., & Speelman, E. (2008). Are challenge (ropes) courses an effective tool? A meta-analysis. Journal of Experiential Education, 31(2), 111–135.CrossRefGoogle Scholar
  15. Gillis, H. L., Speelman, E., Linville, N., Bailey, E., Kalle, A., Oglesbee, N., et al. (2016). Meta-analysis of treatment outcomes measured by the Y-OQ and Y-OQ SR comparing wilderness and non-wilderness treatment programs. Child & Youth Care Forum, 45(6), 851–863. Scholar
  16. Gironda, M. A. (2000). A validity study of the Youth-Outcome Questionnaire and the Ohio Scales. Dissertation Abstracts International: Section B: The Sciences & Engineering, 61(5-B).Google Scholar
  17. Goldberg, S. B., Babins-Wagner, R., Rousmaniere, T., Berzins, S., Hoyt, W. T., Whipple, J. L., et al. (2016). Creating a climate for therapist improvement: A case study of an agency focused on outcomes and deliberate practice. Psychotherapy, 53(3), 367–375. Scholar
  18. Goodman, J. D., McKay, J. R., & DePhilippis, D. (2013). Progress monitoring in mental health and addiction treatment: A means of improving care. Professional Psychology: Research and Practice, 44(4), 231–246. Scholar
  19. Greco, L., Baer, R. A., & Smith, G. T. (2011). Assessing mindfulness in children and adolescents: Development and validation of the child and adolescent mindfulness measure (CAMM). Psychological Assessment, 23, 606–614.CrossRefPubMedGoogle Scholar
  20. Harris, N., Brazeau, J. N. R., Rawana, E. P., Brownlee, K., & Mazmanian, D. (2016). A preliminary examination of a strengths-based treatment for adolescent substance use issues. Canadian Journal of Counselling and Psychotherapy (Online), 50(2), 145–163.Google Scholar
  21. Hatfield, D., McCullough, L., Frantz, S. H., & Krieger, K. (2010). Do we know when our clients get worse? An investigation of therapists’ ability to detect negative client change. Clinical Psychology and Psychotherapy, 17, 25–32. Scholar
  22. Hattie, J., Marsh, H. W., Neill, J. T., & Richards, G. E. (1997). Adventure education and outward bound: Out-of-class experiences that make a lasting difference. Review of Educational Research, 67, 43–87.CrossRefGoogle Scholar
  23. Howard, K. I., Moras, K., Brill, P. L., Martinovich, Z., & Lutz, W. (1996). Evaluation of psychotherapy: Efficacy, effectiveness, and patient progress. American Psychologist, 51(10), 1059–1064.CrossRefPubMedGoogle Scholar
  24. IBM. (2017). IBM: How to cite IBM SPSS statistics or earlier versions of SPSS. Accessed June 21, 2017.
  25. Jacobsen, N. S., & Truax, P. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19.CrossRefGoogle Scholar
  26. Lambert, M. J. (2010). Prevention of treatment failure: The use of measuring, monitoring, and feedback in clinical practice. Washington, DC: American Psychological Association.CrossRefGoogle Scholar
  27. Lambert, M. J. (2017). Maximizing psychotherapy outcome beyond evidence-based medicine. Psychotherapy and Psychosematics, 86, 80–89. Scholar
  28. Lambert, M. J., & Shimokawa, K. (2011). Collecting client feedback. Psychotherapy, 48(1), 72–79. Scholar
  29. Lambert, M. J., Whipple, J. L., Hawkins, E. J., Vermeersch, D. A., Nielsen, S. L., & Smart, D. W. (2003). Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology: Science and Practice, 10(3), 288–301.Google Scholar
  30. Lee, B. R., & Barth, R. (2011). Defining group care programs: An index of reporting standards. Child & Youth Care Forum, 40, 253–266. Scholar
  31. Levin, M. E., Dalrymple, K., & Zimmerman, M. (2014). Which facets of mindfulness predict the presence of substance use disorders in an outpatient psychiatric sample? Psychology of Addictive Behaviors, 28(2), 498–506.CrossRefPubMedGoogle Scholar
  32. Miller, S. D., & Hubble, M. (2011). The road to mastery. Psychotherapy Networker, 35, 22–60.Google Scholar
  33. Miller, S. D., Hubble, M. A., Chow, D., & Seidel, J. (2015). Beyond measures and monitoring: Realizing the potential of feedback-informed treatment. Psychotherapy, 52(4), 449–457.CrossRefPubMedGoogle Scholar
  34. Park, N. (2009). Building strengths of character: Keys to positive youth development. Reclaiming Children and Youth, 18, 42–47.Google Scholar
  35. Patton, M. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage Publications.Google Scholar
  36. Posten, J. M., & Hanson, W. E. (2010). Meta-analysis of psychological assessment as a therapeutic intervention. Psychological Assessment, 22(2), 203–212. Scholar
  37. Russell, K. C., & Farnum, J. (2004). A concurrent model of the wilderness therapy process. Journal of Adventure Education and Outdoor Learning, 4, 39–55.CrossRefGoogle Scholar
  38. Russell, K. C., & Gillis, H. L. (2017). The Adventure Therapy Experience Scale (ATES): The psychometric properties of a scale to measure the unique factors moderating an adventure therapy experience. Journal of Experiential Education. Scholar
  39. Russell, K. C., Gillis, H. L., & Heppner, W. (2015). An examination of mindfulness-based experiences through adventure in substance use disorder treatment for young adult males: A pilot study. Mindfulness, 7(2), 320–328. Scholar
  40. Russell, K. C., Gillis, H. L., & Kivligan, D. (2017, August). Process factors explaining psycho-social outcomes in adventure therapy. Psychotherapy,
  41. Shimokawa, K., Lambert, M. J., & Smart, D. W. (2010). Enhancing treatment outcome of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting and Clinical Psychology, 78(3), 298–311. Scholar
  42. Solstad, S. M., Castonguay, L. G., & Moltu, C. (2017). Patients’ experiences with routine outcome monitoring and clinical feedback systems: A systematic review and synthesis of qualitative empirical literature. Psychotherapy Research. Scholar
  43. Sundet, R. (2012). Therapist perspectives on the use of feedback on process and outcome: Patient-focused research in practice. Canadian Psychology/Psychologie Canadienne, 53(2), 122–130. Scholar
  44. Toback, R. L., Graham-Bermann, S. A., & Paresh, D. P. (2016). Outcomes of a character strengths–based intervention on self-esteem and self-efficacy of psychiatrically hospitalized youths. Psychiatric Services, 67(5), 574–577.CrossRefPubMedGoogle Scholar
  45. Valenstein, M., Adler, D. A., Berlant, J., Dixon, L. B., Dulit, R. A., Goldman, B., et al. (2009). Implementing standardized assessments in clinical care: Now’s the time. Psychiatric Services, 60, 1372–1375. Scholar
  46. van Sonsbeek, M. A. M. S., Hutschemaekers, G. G. J. M., Veerman, J. W., & Tiemens, B. B. G. (2014). Effective components of feedback from routine outcome monitoring (ROM) in youth mental health care: Study protocol of a three-arm parallel-group randomized controlled trial. BMC Psychiatry, 14(1), 3. Scholar
  47. Weems, C. F. (2011). Guidelines for empirical papers on group care programs. Child & Youth Care Forum, 40, 251. Scholar
  48. Wilson, S. J., & Lipsey, M. W. (2000). Wilderness challenge programs for delinquent youth: A meta-analysis of outcome evaluations. Evaluation and Program Planning, 23, 1–12.CrossRefGoogle Scholar
  49. Winters, K. C., & Henly, G. A. (1989). Personal experience inventory and manual. Los Angeles: Western Psychological Services.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Western Washington UniversityBellinghamUSA
  2. 2.Georgia CollegeMilledgevilleUSA
  3. 3.ENVIROSCalgaryCanada

Personalised recommendations