Clinically Significant and Reliable Change: Comparing an Evidence-based Intervention to Usual Care

  • Maria Michelle VardanianEmail author
  • Amrita Ramakrishnan
  • Sarah Peralta
  • Yasmin Siddiqui
  • Suniti P. Shah
  • Elysha Clark-Whitney
  • Anil Chacko
Original Paper



Multiple Family Groups (MFG) is an evidence-based behavioral parent training developed with a specific focus on increasing engagement and decreasing treatment barriers for families of children with disruptive behavior problems within high-risk communities. Previous studies have demonstrated the effectiveness of MFG in improving oppositional behavior at the group-level compared to services as usual (SAU). However, information is lacking regarding intervention effectiveness on an individual-level (i.e., clinical significance).


The reliable change index and clinical cutoff score method was utilized to determine whether MFG produced clinically meaningful changes compared to SAU for both child- and parent-level outcomes in a sample of 320 youth aged 7 to 11-years-old.


A significantly greater percentage of children in the MFG group experienced clinically meaningful change in problem behaviors compared to the SAU group, (p = 0.003, 95% [CI]: 1.610–18.481). A significantly greater number of parents in MFG also demonstrated clinically meaningful change in parental experience of stress compared to SAU, (p = 0.01, 95% [CI]: 1.255–14.704).


Findings suggested clinically significant and reliable improvements in child problem behaviors and decreases in parental perceived stress for families in MFG compared to SAU. Nevertheless, analyses demonstrated that both MFG and SAU resulted in few families obtaining clinically significant or reliable change in their functioning. Ongoing assessments and deeper understanding of intervention effect are needed to better service families in need. Both group- and individual-level comparisons should be considered when examining the effects of a treatment as they may provide a nuanced understanding of evidence-based interventions.


Clinical significance Behavioral parent training Disruptive behavior disorders Usual care Child functioning 


Author Contributions

M.M.V.: Led paper development, data analysis, and writing of the manuscript. A.R.: Collaborated with writing of the manuscript, assisted in data analysis. S.P.: Collaborated with writing of the manuscript, wrote part of the introduction. Y.S.: Assisted in data analysis, wrote part of the results. S.P.S. and E.C.: Collaborated in writing and editing of the manuscript. A.C.: Designed and executed main study, collaborated in editing of final manuscript.

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee (IRB approval received by the Mount Sinai School of Medicine) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed Consent

Informed consent was obtained from all individual participants included in the original study.


  1. Abidin, R. R., & Brunner, J. F. (1995). Development of a parenting alliance inventory. Journal of Clinical Child Psychology, 24(1), 31–40.CrossRefGoogle Scholar
  2. Acri, M., Chacko, A., Gopalan, G., & McKay, M. (2018). Engaging families in treatment for child behavior disorders: a synthesis of the literature. In J. E. Lochman & W. Matthys (Eds), The Wiley handbook of disruptive and impulse-control disorders (pp. 393–409). Hoboken, NJ: John Wiley & Sons Inc.Google Scholar
  3. Adams, J. F. (2001). Impact of parent training on family functioning. Child & Family Behavior Therapy, 23(1), 29–42.CrossRefGoogle Scholar
  4. Allan, C., & Chacko, A. (2018). Adverse events in behavioral parent training for children with ADHD: An under-appreciated phenomenon. The ADHD Report, 26(1), 4–9.CrossRefGoogle Scholar
  5. American Psychiatric Association. (2013). Diagnostic and Statistical Manual of Mental Disorders (DSM-5th ed.). Arlington, VA: American Psychiatric Publishing.CrossRefGoogle Scholar
  6. Atkins, D. C., Bedics, J. D., Mcglinchey, J. B., & Beauchaine, T. P. (2005). Assessing clinical significance: does it matter which method we use? Journal of Consulting and Clinical Psychology, 73(5), 982–989.PubMedCrossRefPubMedCentralGoogle Scholar
  7. Barkham, M., Connell, J., Stiles, W. B., Miles, J. N., Margison, F., Evans, C., & Mellor-Clark, J. (2006). Dose-effect relations and responsive regulation of treatment duration: the good enough level. Journal of Consulting and Clinical Psychology, 74(1), 160–167.PubMedCrossRefPubMedCentralGoogle Scholar
  8. Bellamy, J. L., Gopalan, G., & Traube, D. E. (2010). A national study of the impact of outpatient mental health services for children in long-term foster care. Clinical Child Psychology and Psychiatry, 15(4), 467–479.PubMedPubMedCentralCrossRefGoogle Scholar
  9. Bickman, L. (2008). Why don’t we have effective mental health services? Administration and Policy in Mental Health and Mental Health Services Research, 35(6), 437–439.PubMedCrossRefPubMedCentralGoogle Scholar
  10. Bickman, L. (2000). The most dangerous and difficult question in mental health services research. Mental Health Services Research, 2(2), 71–72.CrossRefGoogle Scholar
  11. Campbell, T. C. (2005). An introduction to clinical significance: an alternative index of intervention effect for group experimental designs. Journal of Early Intervention, 27(3), 210–227.CrossRefGoogle Scholar
  12. Chacko, A., Alan, C., Uderman, J., Cornwell, M., Anderson, L., & Chimiklis, A. (2015a). Training parents of children with ADHD. In R. Barkley (Ed.), Attention deficit hyperactivity disorder: a handbook for diagnosis and treatment (pp. 513–536). New York, NY: Guilford Press.Google Scholar
  13. Chacko, A., Fabiano, G. A., Doctoroff, G. L., & Fortson, B. (2018a). Engaging fathers in effective parenting for preschool children using shared book reading: a randomized controlled trial. Journal of Clinical Child & Adolescent Psychology, 47(1), 79–93.CrossRefGoogle Scholar
  14. Chacko, A., Gopalan, G., Franco, L., Dean-Assael, K., Jackson, J., Marcus, S., & McKay, M. (2015b). Multiple family group service model for children with disruptive behavior disorders. Journal of Emotional and Behavioral Disorders, 23(2), 67–77.PubMedCrossRefGoogle Scholar
  15. Chacko, A., Granski, M., Horn, E. P., Levy, M. D., Dahl, V., Lacks, R. S., & Ramakrishnan, A. (2018b). Prevention of disruptive behavior problems in children. In M. M. Martel (Ed.), Developmental pathways to disruptive, impulse-control and conduct disorders (pp. 347–380). London, UK: Academic Press.CrossRefGoogle Scholar
  16. Chacko, A., Isham, A., Cleek, A. F., & McKay, M. M. (2016a). Using mobile health technology to improve behavioral skill implementation through homework in evidence-based parenting intervention for disruptive behavior disorders in youth: study protocol for intervention development and evaluation. Pilot and Feasibility Studies, 2(1), 57–68.PubMedPubMedCentralCrossRefGoogle Scholar
  17. Chacko, A., Jensen, S. A., Lowry, L. S., Cornwell, M., Chimklis, A., Chan, E., & Pulgarin, B. (2016b). Engagement in behavioral parent training: review of the literature and implications for practice. Clinical Child and Family Psychology Review, 19(3), 204–215.PubMedCrossRefGoogle Scholar
  18. Chacko, A., & Scavenius, C. (2018). Bending the curve: a community-based behavioral parent training model to address ADHD-related concerns in the voluntary sector in Denmark. Journal of Abnormal Child Psychology, 46(3), 505–517.PubMedCrossRefGoogle Scholar
  19. Chacko, A., Scavenius, C., Rewitz, A. F., & Lydiksen, C. L. (2018c). Caring in chaos: a behavioral parent training model for resource-limited community providers. The ADHD Report, 26(4), 1–8.CrossRefGoogle Scholar
  20. Chacko, A., Uderman, J. Z., & Zwilling, A. (2013). Lessons learned in enhancing behavioral parent training for high-risk families of youth with ADHD. The ADHD Report, 21(4), 6–11.PubMedPubMedCentralCrossRefGoogle Scholar
  21. Chacko, A., Wymbs, B. T., Chimiklis, A., Wymbs, F. A., & Pelham, W. E. (2012). Evaluating a comprehensive strategy to improve engagement to group-based behavioral parent training for high-risk families of children with ADHD. Journal of Abnormal Child Psychology, 40(8), 1351–1362.PubMedCrossRefPubMedCentralGoogle Scholar
  22. Chacko, A., Wymbs, B., Rajwan, E., Wymbs, F., & Feirsen, N. (2017). Characteristics of parents of children with ADHD who never attend, drop out, and complete behavioral parent training. Journal of Child and Family Studies, 26(3), 950–960.CrossRefGoogle Scholar
  23. Chacko, A., Wymbs, B. T., Wymbs, F. A., Pelham, W. E., Swanger-Gagne, M. S., Girio, E., & O’Connor, B. (2009). Enhancing traditional behavioral parent training for single mothers of children with ADHD. Journal of Clinical Child & Adolescent Psychology, 38(2), 206–218.CrossRefGoogle Scholar
  24. Chavis, A., Hudnut-Beumler, J., Webb, M. W., Neely, J. A., Bickman, L., Dietrich, M. S., & Scholer, S. J. (2013). A brief intervention affects parents’ attitudes toward using less physical punishment. Child Abuse & Neglect, 37(12), 1192–1201.CrossRefGoogle Scholar
  25. Chronis, A., Chacko, A., Fabiano, G., Wymbs, B., & Pelham, Jr, W. (2004). Enhancements to the behavioral parent training paradigm for families of children with ADHD: Review and future directions. Clinical Child and Family Psychology Review, 7(1), 1–27.PubMedCrossRefGoogle Scholar
  26. Dadds, M. R., Maujean, A., & Fraser, J. A. (2003). Parenting and conduct problems in children: Australian data and psychometric properties of the Alabama parenting questionnaire. Australian Psychologist, 38(3), 238–241.CrossRefGoogle Scholar
  27. Deater-Deckard, K. (2008). Parenting stress. New Haven, CT: Yale University Press.Google Scholar
  28. Deater-Deckard, K. D. & Panneton, R. (Eds) (2017). Parental stress and early child development. Cham, CH: Springer.Google Scholar
  29. Devilly, G.J. (2005). ClinTools Software for Windows: Version 4 (computer programme). Psytek Ltd.
  30. Eyberg, S. M., Nelson, M. M., & Boggs, S. R. (2008). Evidence-based psychosocial treatments for children and adolescents with disruptive behavior. Journal of Clinical Child & Adolescent Psychology, 37(1), 215–237.CrossRefGoogle Scholar
  31. Fabiano, G. A., Pelham, W. E., Coles, E. K., Gnagy, E. M., Chronis-Tuscano, A., & O’Connor, B. C. (2009). A meta-analysis of behavioral treatments for attention-deficit/hyperactivity disorder. Clinical Psychology Review, 29(2), 129–140.PubMedCrossRefPubMedCentralGoogle Scholar
  32. Fabiano, G. A., Pelham, J., William, E., Waschbusch, D. A., Gnagy, E. M., Lahey, B. B., & Burrows-MacLean, L. (2006). A practical measure of impairment: psychometric properties of the impairment rating scale in samples of children with attention deficit hyperactivity disorder and two school-based samples. Journal of Clinical Child & Adolescent Psychology, 35(3), 369–385.CrossRefGoogle Scholar
  33. Ferrer, R., & Pardo, A. (2014). Clinically meaningful change: false positives in the estimation of individual change. Psychological Assessment, 26(2), 370–383.PubMedCrossRefPubMedCentralGoogle Scholar
  34. Frazier, S. L., Chacko, A., Van Gessel, C., O’Boyle, C., & Pelham, W. E. (2012). The summer treatment program meets the south side of Chicago: bridging science and service in urban after‐school programs. Child and Adolescent Mental Health, 17(2), 86–92.PubMedCrossRefPubMedCentralGoogle Scholar
  35. Garland, A. F., Bickman, L., & Chorpita, B. F. (2010). Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health and Mental Health Services Research, 37(1–2), 15–26.PubMedCrossRefPubMedCentralGoogle Scholar
  36. Gerdes, A. C., Haack, L. M., & Schneider, B. W. (2012). Parental functioning in families of children with ADHD: evidence for behavioral parent training and importance of clinically meaningful change. Journal of Attention Disorders, 16(2), 147–156.PubMedCrossRefPubMedCentralGoogle Scholar
  37. Gopalan, G., Bornheimer, L. A., Acri, M., Winters, A., O’Brien, K. H., & Chacko, A. (2018). Multiple family group service delivery model for children with disruptive behavior disorders: Impact on caregiver stress and depressive symptoms. Journal of Emotional and Behavioral Disorders, 26(3), 182–192.PubMedCrossRefPubMedCentralGoogle Scholar
  38. Gopalan, G., Chacko, A., Franco, L., Dean-Assael, K. M., Rotko, L. E., Marcus, S. M., & McKay, M. M. (2015a). Multiple family groups for children with disruptive behavior disorders: Child outcomes at 6-month follow-up. Journal of Child and Family Studies, 24(9), 2721–2733.PubMedCrossRefPubMedCentralGoogle Scholar
  39. Gopalan, G., Fuss, A., & Wisdom, J. P. (2015b). Multiple family groups for child behavior difficulties retention among child welfare-involved caregivers. Research on Social Work Practice, 25(5), 564–577.PubMedCrossRefGoogle Scholar
  40. Gopalan, G., Small, L., Fuss, A., Bowman, M., Jackson, J., Marcus, S., & Chacko, A. (2015c). Multiple family groups to reduce child disruptive behavior difficulties: Moderating effects of child welfare status on child outcomes. Child Abuse & Neglect, 46, 207–219.CrossRefGoogle Scholar
  41. Gresham, F. M. (2015). Disruptive behavior disorders: evidence-based practice for assessment and interventions. New York, NY: Guilford Press.Google Scholar
  42. Haskett, M. E., Ahern, L. S., Ward, C. S., & Allaire, J. C. (2006). Factor structure and validity of the parenting stress-index-short form. Journal of Clinical Child & Adolescent Psychology, 35(2), 302–312.CrossRefGoogle Scholar
  43. Health Resources and Services Administration [HRSA] (2016). National projections of supply and demand for selected behavioral health practitioners: 2013–2025. Department of Health and Human Services. Accessed 23 Jan 2019.
  44. Heath, C. L., Curtis, D. F., Fan, W., & McPherson, R. (2015). The association between parenting stress, parenting self-efficacy, and the clinical significance of child ADHD symptom change following behavior therapy. Child Psychiatry & Human Development, 46(1), 118–129.CrossRefGoogle Scholar
  45. Hoagwood, K., Cavaleri, M., Olin, S., Burns, B., Slaton, E., Gruttadaro, D., & Hughes, R. (2010). Family support in children’s mental health: a review and synthesis. Clinical Child and Family Psychology Review, 13(1), 1–45.PubMedCrossRefPubMedCentralGoogle Scholar
  46. Hoagwood, K., & Kolko, D. J. (2009). Introduction to the special section on practice contexts: A glimpse into the nether world of public mental health services for children and families. Administration and Policy in Mental Health and Mental Health Services Research, 36(1), 35–36.PubMedCrossRefPubMedCentralGoogle Scholar
  47. IBM Corp. (2017). IBM SPSS statistics for windows, version25.0. Armonk, NY: IBM Corp. Released.Google Scholar
  48. Jacobson, N. S., & Truax, P. (1991). Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19.PubMedCrossRefPubMedCentralGoogle Scholar
  49. Kaminski, J. W., & Claussen, A. H. (2017). Evidence base update for psychosocial treatment of disruptive behaviors in children. Journal of Clinical Child and Adolescent Psychology, 46(4), 477–499.PubMedPubMedCentralCrossRefGoogle Scholar
  50. Karpenko, V., Owens, J. S., Evangelista, N. M., & Dodds, C. (2009). Clinically significant symptom change in children with attention-deficit/hyperactivity disorder: Does it correspond with reliable improvement in functioning? Journal of Clinical Psychology, 65(1), 76–93.PubMedCrossRefGoogle Scholar
  51. Kazdin, A. E. (1995). Child, parent and family dysfunction as predictors of outcome in cognitive-behavioral treatment of antisocial children. Behaviour Research and Therapy, 33(3), 271–281.PubMedCrossRefGoogle Scholar
  52. Kazdin, A. E., & Whitley, M. K. (2003). Treatment of parental stress to enhance therapeutic change among children referred for aggressive and antisocial behavior. Journal of Consulting and Clinical Psychology, 71(3), 504–515.CrossRefGoogle Scholar
  53. McCart, M. R., Priester, P. E., Davies, W. H., & Azen, R. (2006). Differential effectiveness of behavioral parent-training and cognitive-behavioral therapy for antisocial youth: a meta-analysis. Journal of Abnormal Child Psychology, 34(4), 525–541.CrossRefGoogle Scholar
  54. McKay, M. M., Gonzales, J., Quintana, E., Kim, L., & Abdul-Adil, J. (1999). Multiple family groups: An alternative for reducing disruptive behavioral difficulties of urban children. Research on Social Work Practice, 9(5), 593–607.CrossRefGoogle Scholar
  55. Newnham, E. A., & Page, A. C. (2010). Bridging the gap between best evidence and best practice in mental health. Clinical Psychology Review, 30(1), 127–142.PubMedCrossRefGoogle Scholar
  56. Ogles, B. M., Lunnen, K. M., & Bonesteel, K. (2001). Clinical significance: history, application, and current practice. Clinical Psychology Review, 21(3), 421–446.PubMedCrossRefGoogle Scholar
  57. Oruche, U. M., Draucker, C. B., Robb, S. L., Aalsma, M. C., Pescosolido, B. A., & Brown-Podgorski, B. (2017). Developing a multiple caregiver group for caregivers of adolescents with disruptive behaviors. Archives of Psychiatric Nursing, 31(6), 596–601.PubMedCrossRefGoogle Scholar
  58. Owens, J. S., Johannes, L. M., & Karpenko, V. (2009). The relation between change in symptoms and functioning in children with ADHD receiving school-based mental health services. School Mental Health, 1(4), 183–195.CrossRefGoogle Scholar
  59. Pelham, W. E., & Fabiano, G. A. (2008). Evidence-based psychosocial treatments for attention-deficit/hyperactivity disorder. Journal of Clinical Child & Adolescent Psychology, 37(1), 184–214.CrossRefGoogle Scholar
  60. Pelham, W., Gnagy, E., Greenslade, K., & Milich, R. (1992). Teacher ratings of DSM-III-R symptoms for the disruptive behavior disorder. Journal of The American Academy of Child & Adolescent Psychiatry, 31(2), 210–218.CrossRefGoogle Scholar
  61. Rajwan, E., Chacko, A., Wymbs, B., & Wymbs, F. (2014). Evaluating clinically significant change in mother and child functioning: comparison of traditional and enhanced behavioral parent training. Journal of Abnormal Child Psychology, 42(8), 1407–1412.PubMedCrossRefPubMedCentralGoogle Scholar
  62. Sheldrick, R. C., Kendall, P. C., & Heimberg, R. G. (2001). The clinical significance of treatments: a comparison of three treatments for conduct disordered children. Clinical Psychology: Science and Practice, 8(4), 418–430.Google Scholar
  63. Shelton, K. K., Frick, P. J., & Wootton, J. (1996). Assessment of parenting practices in families of elementary school-age children. Journal of Clinical Child Psychology, 25(3), 317–329.CrossRefGoogle Scholar
  64. Stoltz, S., Deković, M., van Londen, M., Orobio de Castro, B., & Prinzie, P. (2013). What works for whom, how and under what circumstances? Testing moderated mediation of intervention effects on externalizing behavior in children. Social Development, 22(2), 406–425.CrossRefGoogle Scholar
  65. Warren, J. S., Nelson, P. L., & Burlingame, G. M. (2009). Identifying youth at risk for treatment failure in outpatient community mental health services. Journal of Child and Family Studies, 18(6), 690–701.CrossRefGoogle Scholar
  66. Warren, J. S., Nelson, P. L., Mondragon, S. A., Baldwin, S. A., & Burlingame, G. M. (2010). Youth psychotherapy change trajectories and outcomes in usual care: community mental health versus managed care settings. Journal of Consulting and Clinical Psychology, 78(2), 144–155.PubMedCrossRefPubMedCentralGoogle Scholar
  67. Waschbusch, D., & Willoughby, M. (2008). Parent and teacher ratings on the IOWA Connors Rating Scale. Journal of Psychopathology and Behavioral Assessment, 30(3), 180–192.CrossRefGoogle Scholar
  68. Weeland, J., Chhangur, R. R., van der Giessen, D., Matthys, W., de Castro, B. O., & Overbeek, G. (2017). Intervention effectiveness of The Incredible Years: new insights into sociodemographic and intervention-based moderators. Behavior Therapy, 48(1), 1–18.PubMedCrossRefGoogle Scholar
  69. Weisz, J. R. (2004). Psychotherapy for children and adolescents: Evidence-based treatments and case examples. Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
  70. Weisz, J. R., Kuppens, S., Ng, M. Y., Eckshtain, D., Ugueto, A. M., Vaughn-Coaxum, R., & Weersing, V. R. (2017). What five decades of research tells us about the effects of youth psychological therapy: a multilevel meta-analysis and implications for science and practice. American Psychologist, 72(2), 79–117.PubMedCrossRefPubMedCentralGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.New York UniversityNew YorkUSA

Personalised recommendations