Advertisement

The Child and Adolescent Services Assessment: Interrater Reliability and Predictors of Rater Disagreement

  • Karen T. G. Schwartz
  • Amanda A. Bowling
  • John F. Dickerson
  • Frances L. Lynch
  • David A. Brent
  • Giovanna Porta
  • Satish Iyengar
  • V. Robin Weersing
Original Article

Abstract

The current study evaluated the interrater reliability of the Child and Adolescent Services Assessment (CASA), a widely used structured interview measuring pediatric mental health service use. Interviews (N = 72) were randomly selected from a pediatric effectiveness trial, and audio was coded by an independent rater. Regressions were employed to identify predictors of rater disagreement. Interrater reliability was high for items (> 94%) and summary metrics (ICC > .79) across service sectors. Predictors of disagreement varied by domain; significant predictors indexed higher clinical severity or social disadvantage. Results support the CASA as a reliable and robust assessment of pediatric service use, but administrators should be alert when assessing vulnerable populations.

Keywords

Reliability Service use Child Adolescent 

Notes

Funding

This study was funded by the National Institute of Mental Health (5R01MH084935; 5R01MH084916); however, the funding source had no role in any aspect of study creation, implementation, or preparation of this manuscript.

Compliance with Ethical Standards

Conflict of interest

Dr. Brent receives research support from NIMH, royalties from Guilford Press, royalties from the electronic self-rated version of the C-SSRS from ERT, Inc., royalties from performing duties as an UptoDate Psychiatry Section Editor, and consulting fees from Healthwise. He has no other conflicts of interest to report. Ms. Schwartz, Ms. Bowling, Dr. Dickerson, Dr. Lynch, Ms. Porta, Dr. Iyengar and Dr. Weersing declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.

Informed Consent

Informed consent and assent were obtained from all adult and youth participants, respectively, included in the study, in accordance with the San Diego State University and University of California, San Diego institutional review boards.

References

  1. Angold, A., Costello, E. J., Farmer, E. M. Z., Burns, B. J., & Erkanli, A. (1999). Impaired but undiagnosed. Journal of the American Academy of Child & Adolescent Psychiatry, 38, 129–137.  https://doi.org/10.1097/00004583-199902000-00011.CrossRefGoogle Scholar
  2. Angold, A., Costello, E. J., Pickles, A., Winder, F., & Silver, D. (1987). The development of a questionnaire for use in epidemiological studies of depression in children and adolescents. Unpublished manuscript, London University.Google Scholar
  3. Ascher, B. H., Farmer, E. M. Z., Burns, B. J., & Angold, A. (1996). The child and adolescent services assessment (CASA): Description and psychometrics. Journal of Emotional and Behavioral Disorders, 4, 12–20.  https://doi.org/10.1177/106342669600400102.CrossRefGoogle Scholar
  4. Beardslee, W. R., Brent, D. A., Weersing, V. R., Clarke, G. N., Porta, G., Hollon, S. D.,… Garber, J. (2013). Prevention of depression in at-risk adolescents: Longer-term effects. JAMA Psychiatry, 70, 1161–1170.  https://doi.org/10.1001/jamapsychiatry.2013.295.CrossRefPubMedPubMedCentralGoogle Scholar
  5. Birmaher, B., Khetarpal, S., Brent, D., Cully, M., Balach, L., Kaufman, J. & Neer, S. M. (1997). The screen for child anxiety related emotional disorders (SCARED): Scale construction and psychometric characteristics. Journal of the American Academy of Child & Adolescent Psychiatry, 36, 545–553.  https://doi.org/10.1097/00004583-199704000-00018.CrossRefGoogle Scholar
  6. Burns, B. J., Angold, A., & Costello, E. J. (1992). Measuring child, adolescent, and family service use. In L. Bickman & D. J. Rog (Eds.), Evaluating mental health services for children (pp. 17–29). San Francisco, CA: Jossey-Bass.Google Scholar
  7. Burns, B. J., Costello, E. J., Angold, A., Tweed, D., Stangle, D., Farmer, E. M. Z., & Erkanli, A. (1995). Children’s mental health service use across service sectors. Health Affairs, 14, 147–159.  https://doi.org/10.1377/hlthaff.14.3.147.CrossRefPubMedGoogle Scholar
  8. Burns, B. J., Phillips, S. D., Wagner, H. R., Barth, R. P., Kolko, D. J., Campbell, Y., & Landsverk, J. (2004). Mental health need and access to mental health services by youths involved with child welfare: A national survey. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 960–970.  https://doi.org/10.1097/01.chi.0000127590.95585.65.CrossRefGoogle Scholar
  9. Bussing, R., Mason, D. M., Leon, C. E., & Sinha, K. (2003). Agreement between CASA parent reports and provider records of children’s ADHD services. The Journal of Behavioral Health Services & Research, 30, 462–469.  https://doi.org/10.1007/BF02287433.CrossRefGoogle Scholar
  10. Corral, I., & Landrine, H. (2010). Methodological and statistical issues in research with diverse samples: The problem of measurement equivalence. In H. Landrine & N. F. Russo (Eds.), Handbook of diversity in feminist psychology (pp. 83–121). New York: Springer.Google Scholar
  11. Costello, E. J., Angold, A., Burns, B. J., Erkanli, A., Stangl, D. K., & Tweed, D. L. (1996). The great smoky mountains study of youth: Functional impairment and serious emotional disturbance. Archives of General Psychiatry, 53, 1137–1143.  https://doi.org/10.1001/archpsyc.1996.01830120077013.CrossRefPubMedGoogle Scholar
  12. Costello, E. J., Burns, B. J., Costello, A. J., Edelbrock, C., Dulcan, M., & Brent, D. (1988). Service utilization and psychiatric diagnosis in pediatric primary care: The role of the gatekeeper. Pediatrics, 82, 435–441.PubMedGoogle Scholar
  13. Costello, E. J., He, J., Sampson, N. A., Kessler, R. C., & Merikangas, K. R. (2014). Services for adolescents with psychiatric disorders: 12-month data from the National Comorbidity Survey-Adolescent. Psychiatric Services, 65, 359–366.  https://doi.org/10.1176/appi.ps.201100518.CrossRefPubMedGoogle Scholar
  14. Domino, M. E., Burns, B. J., Silva, S. G., Kratochvil, C. J., Vitiello, B., Reinecke, M. A.,… March, J. S. (2008). Cost-effectiveness of treatments for adolescent depression: Results from TADS. American Journal of Psychiatry, 165, 588–596.  https://doi.org/10.1176/appi.ajp.2008.07101610.CrossRefPubMedGoogle Scholar
  15. Domino, M. E., Foster, E. M., Vitiello, B., Kratochvil, C. J., Burns, B. J., Silva, S. G.,… March, J. S. (2009). Relative cost-effectiveness of treatments for adolescent depression: 36-Week results from the TADS randomized trial. Journal of the American Academy of Child & Adolescent Psychiatry, 48, 711–720.  https://doi.org/10.1097/CHI.0b013e3181a2b319.CrossRefGoogle Scholar
  16. Farmer, E. M. Z., Angold, A., Burns, B. J., & Costello, E. J. (1994). Reliability of self-reported service use: Test-retest consistency of children’s responses to the Child and Adolescent Services Assessment (CASA). Journal of Child and Family Studies, 3, 307–325.  https://doi.org/10.1007/BF02234688.CrossRefGoogle Scholar
  17. Farmer, E. M. Z., Burns, B. J., Angold, A., & Costello, E. J. (1997). Impact of children’s mental health problems on families: Relationships with service use. Journal of Emotional and Behavioral Disorders, 5, 230–238.  https://doi.org/10.1177/106342669700500406.CrossRefGoogle Scholar
  18. Farmer, E. M. Z., Mustillo, S. A., Wagner, H. R., Burns, B. J., Kolko, D. J., Barth, R. P., & Leslie, L. K. (2010). Service use and multi-sector use for mental health problems by youth in contact with child welfare. Children and Youth Services Review, 32, 815–821.  https://doi.org/10.1016/j.childyouth.2010.01.019.CrossRefGoogle Scholar
  19. Farmer, E. M. Z., Stangl, D. K., Burns, B. J., Costello, E. J., & Angold, A. (1999). Use, persistence, and intensity: Patterns of care for children’s mental health across one year. Community Mental Health Journal, 35, 31–46.CrossRefGoogle Scholar
  20. Garber, J., Clarke, G. N., Weersing, V. R., Beardslee, W. R., Brent, D. A., Gladstone, T. R. G.,… Iyengar, S. (2009). Prevention of depression in at-risk adolescents: A randomized controlled trial. JAMA, 301, 2215–2224.CrossRefGoogle Scholar
  21. Geisinger, K. F. (2013). Reliability. In K. F. Geisinger (Ed.), APA handbook of testing and assessment in psychology: Vol. 1. Test theory and testing and assessment in industrial and organizational psychology (pp. 21–42). Washington, DC: American Psychological Association.Google Scholar
  22. Ghandour, R. M., Kogan, M. D., Blumberg, S. J., Jones, J. R., & Perrin, J. M. (2012). Mental health conditions among school-aged children: Geographic and sociodemographic patterns in prevalence and treatment. Journal of Developmental & Behavioral Pediatrics, 33, 42–54.  https://doi.org/10.1097/DBP.0b013e31823e18fd.CrossRefGoogle Scholar
  23. Griffin, J. A., Cicchetti, D., & Leaf, P. J. (1993). Characteristics of youths identified from a psychiatric case register as first-time users of services. Hospital and Community Psychiatry, 44, 62–65.  https://doi.org/10.1176/ps.44.1.62.CrossRefPubMedGoogle Scholar
  24. Guy, W. (1976). Clinical global impression scale. Rockville, MD: National Institute of Mental Health.Google Scholar
  25. Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23–34.CrossRefGoogle Scholar
  26. Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002). The psychotherapy dose-response effect and its implications for treatment delivery services. Clinical Psychology: Science and Practice, 9, 329–343.Google Scholar
  27. Hurlburt, M. S., Leslie, L. K., Landsverk, J., Barth, R. P., Burns, B. J., Gibbons, R. D.,… Zhang, J. (2004). Contextual predictors of mental health service use among children open to child welfare. Archives of General Psychiatry, 61, 1217–1224.  https://doi.org/10.1001/archpsyc.61.12.1217.CrossRefPubMedPubMedCentralGoogle Scholar
  28. Hwang, W., Myers, H. F., Abe-Kim, J., & Ting, J. Y. (2008). A conceptual paradigm for understanding culture’s impact on mental health: The cultural influences on mental health (CIMH) model. Clinical Psychology Review, 28, 211–227.  https://doi.org/10.1016/j.cpr.2007.05.001.CrossRefPubMedGoogle Scholar
  29. Jensen, P. S., Hinshaw, S. P., Swanson, J. M., Greenhill, L. L., Conners, C. K., Arnold, L. E.,… Wigal, T. (2001). Findings from the NIMH Multimodal Treatment Study of ADHD (MTA): Implications and applications for primary care providers. Developmental and Behavioral Pediatrics, 22, 60–73.CrossRefGoogle Scholar
  30. Kazdin, A. E., Holland, L., Crowley, M., & Breton, S. (1997). Barriers to treatment participation scale: Evaluation and validation in the context of child outpatient treatment. Journal of Child Psychology and Psychiatry, 38, 1051–1062.  https://doi.org/10.1111/j.1469-7610.1997.tb01621.x.CrossRefPubMedGoogle Scholar
  31. Kutner, M. H., Nachtsheim, C. J., & Neter, J. (2004). Applied linear regression models (4th edn.). Irwin: McGraw-Hill.Google Scholar
  32. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174.  https://doi.org/10.2307/2529310.CrossRefPubMedPubMedCentralGoogle Scholar
  33. Leslie, L. K., Hurlburt, M. S., James, S., Landsverk, J., Slymen, D. J., & Zhang, J. (2005). Relationship between entry into child welfare and mental health service use. Psychiatric Services, 56, 981–987.  https://doi.org/10.1176/appi.ps.56.8.981.CrossRefPubMedGoogle Scholar
  34. Lynch, F. L., Dickerson, J. F., Clarke, G., Vitiello, B., Porta, G., Wagner, K. D., … Brent, D. (2011). Incremental cost-effectiveness of combined therapy vs medication only for youth with Selective Serotonin Reuptake Inhibitor–resistant depression: Treatment of SSRI-Resistant Depression in Adolescents Trial findings. Archives of General Psychiatry, 68, 253–262.CrossRefGoogle Scholar
  35. Mayr, U., & Kliegl, R. (2000). Task-set switching and long-term memory retrieval. Journal of Experimental Psychology: Learning, Memory, and Cognition, 5, 1124–1140.Google Scholar
  36. Merikangas, K. R., He, J., Burstein, M. E., Swendsen, J., Avenevoli, S., Case, B.,… Olfson, M. (2011). Service utilization for lifetime mental disorders in U.S. adolescents: Results of the National Comorbidity Survey Adolescent supplement (NCS-A). Journal of the American Academy of Child & Adolescent Psychiatry, 50, 32–45.  https://doi.org/10.1016/j.jaac.2010.10.006.CrossRefGoogle Scholar
  37. Nilchaikovit, T., Hill, J. M., & Holland, J. C. (1993). The effects of culture on illness behavior and medical care. Asian and American differences. General Hospital Psychiatry, 15, 41–50.CrossRefGoogle Scholar
  38. Radloff, L. S. (1977). The CES-D scale: A self-report depression scale for research in the general population. Applied Psychological Measurement, 1, 385–401.CrossRefGoogle Scholar
  39. Sanchez, A. L., Cornacchio, D., Poznanski, B., Golik, A. M., Chou, T., & Comer, J. S. (2018). The effectiveness of school-based mental health services for elementary-aged children: A meta-analysis. Journal of the American Academy of Child & Adolescent Psychiatry, 57, 153–165.  https://doi.org/10.1016/j.jaac.2017.11.022.CrossRefGoogle Scholar
  40. Schwarz, N. (1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93–105.  https://doi.org/10.1037//0003-066X.54.2.93.CrossRefGoogle Scholar
  41. Simon, A. E., Pastor, P. N., Reuben, C. A., Huang, L. N., & Goldstrom, I. D. (2015). Use of mental health services by children ages six to 11 with emotional or behavioral difficulties. Psychiatric Services, 66, 930–937.  https://doi.org/10.1176/appi.ps.201400342.CrossRefPubMedGoogle Scholar
  42. Spielberger, C. D., Gorsuch, R. L., Lushene, R., Vagg, P. R., & Jacobs, G. A. (1983). Manual for the state-trait anxiety inventory. Palo Alto, CA: Consulting Psychologists Press.Google Scholar
  43. U.S. Department of Health and Human Services. (2001). Mental health: Culture, race, and ethnicity—a supplement to mental health: A report of the surgeon general. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services.CrossRefGoogle Scholar
  44. Walker, L. S., & Garber, J. (2003). Manual for the children’s somatization inventory. Nashville, TN: Vanderbilt University Medical Center.Google Scholar
  45. Weersing, V. R., Brent, D. A., Rozenman, M. S., Gonzalez, A., Jeffreys, M., Dickerson, J. F.,… Iyengar, S. (2017). Brief behavioral therapy for pediatric anxiety and depression in primary care: A randomized clinical trial. JAMA Psychiatry.  https://doi.org/10.1001/jamapsychiatry.2017.0429.CrossRefPubMedPubMedCentralGoogle Scholar
  46. Weersing, V. R., Gonzalez, A., Campo, J. V., & Lucas, A. N. (2008). Brief behavioral therapy for pediatric anxiety and depression: Piloting an integrated treatment approach. Cognitive and Behavioral Practice, 15, 126–139.CrossRefGoogle Scholar
  47. Weisz, J. R., Jensen-Doss, A., & Hawley, K. M. (2006). Evidence-based youth psychotherapies versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist, 61, 671–689.  https://doi.org/10.1037/0003-066X.61.7.671.CrossRefPubMedGoogle Scholar
  48. Wilk, M. B., & Gnanadesikan, R. (1968). Probability plotting methods for the analysis of data. Biometrika, 55, 1–17.  https://doi.org/10.1093/biomet/55.1.1.CrossRefPubMedGoogle Scholar
  49. Wood, A., Kroll, L., Moore, A., & Harrington, R. (1995). Properties of the mood and feelings questionnaire in adolescent psychiatric outpatients: A research note. Child Psychology & Psychiatry & Allied Disciplines, 36, 327–334.CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Karen T. G. Schwartz
    • 1
  • Amanda A. Bowling
    • 2
  • John F. Dickerson
    • 3
  • Frances L. Lynch
    • 3
  • David A. Brent
    • 4
    • 5
  • Giovanna Porta
    • 5
  • Satish Iyengar
    • 6
  • V. Robin Weersing
    • 1
    • 2
  1. 1.San Diego State University/University of California, San Diego Joint Doctoral Program in Clinical PsychologySan DiegoUSA
  2. 2.Department of PsychologySan Diego State UniversitySan DiegoUSA
  3. 3.Center for Health ResearchKaiser Permanente NorthwestPortlandUSA
  4. 4.Department of PsychiatryUniversity of Pittsburgh School of MedicinePittsburghUSA
  5. 5.Western Psychiatric Institute and ClinicUniversity of Pittsburgh Medical CenterPittsburghUSA
  6. 6.Department of StatisticsUniversity of PittsburghPittsburghUSA

Personalised recommendations