The Child and Adolescent Services Assessment: Interrater Reliability and Predictors of Rater Disagreement
The current study evaluated the interrater reliability of the Child and Adolescent Services Assessment (CASA), a widely used structured interview measuring pediatric mental health service use. Interviews (N = 72) were randomly selected from a pediatric effectiveness trial, and audio was coded by an independent rater. Regressions were employed to identify predictors of rater disagreement. Interrater reliability was high for items (> 94%) and summary metrics (ICC > .79) across service sectors. Predictors of disagreement varied by domain; significant predictors indexed higher clinical severity or social disadvantage. Results support the CASA as a reliable and robust assessment of pediatric service use, but administrators should be alert when assessing vulnerable populations.
KeywordsReliability Service use Child Adolescent
This study was funded by the National Institute of Mental Health (5R01MH084935; 5R01MH084916); however, the funding source had no role in any aspect of study creation, implementation, or preparation of this manuscript.
Compliance with Ethical Standards
Conflict of interest
Dr. Brent receives research support from NIMH, royalties from Guilford Press, royalties from the electronic self-rated version of the C-SSRS from ERT, Inc., royalties from performing duties as an UptoDate Psychiatry Section Editor, and consulting fees from Healthwise. He has no other conflicts of interest to report. Ms. Schwartz, Ms. Bowling, Dr. Dickerson, Dr. Lynch, Ms. Porta, Dr. Iyengar and Dr. Weersing declare that they have no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. This article does not contain any studies with animals performed by any of the authors.
Informed consent and assent were obtained from all adult and youth participants, respectively, included in the study, in accordance with the San Diego State University and University of California, San Diego institutional review boards.
- Angold, A., Costello, E. J., Pickles, A., Winder, F., & Silver, D. (1987). The development of a questionnaire for use in epidemiological studies of depression in children and adolescents. Unpublished manuscript, London University.Google Scholar
- Beardslee, W. R., Brent, D. A., Weersing, V. R., Clarke, G. N., Porta, G., Hollon, S. D.,… Garber, J. (2013). Prevention of depression in at-risk adolescents: Longer-term effects. JAMA Psychiatry, 70, 1161–1170. https://doi.org/10.1001/jamapsychiatry.2013.295.CrossRefPubMedPubMedCentralGoogle Scholar
- Birmaher, B., Khetarpal, S., Brent, D., Cully, M., Balach, L., Kaufman, J. & Neer, S. M. (1997). The screen for child anxiety related emotional disorders (SCARED): Scale construction and psychometric characteristics. Journal of the American Academy of Child & Adolescent Psychiatry, 36, 545–553. https://doi.org/10.1097/00004583-199704000-00018.CrossRefGoogle Scholar
- Burns, B. J., Angold, A., & Costello, E. J. (1992). Measuring child, adolescent, and family service use. In L. Bickman & D. J. Rog (Eds.), Evaluating mental health services for children (pp. 17–29). San Francisco, CA: Jossey-Bass.Google Scholar
- Burns, B. J., Phillips, S. D., Wagner, H. R., Barth, R. P., Kolko, D. J., Campbell, Y., & Landsverk, J. (2004). Mental health need and access to mental health services by youths involved with child welfare: A national survey. Journal of the American Academy of Child & Adolescent Psychiatry, 43, 960–970. https://doi.org/10.1097/01.chi.0000127590.95585.65.CrossRefGoogle Scholar
- Corral, I., & Landrine, H. (2010). Methodological and statistical issues in research with diverse samples: The problem of measurement equivalence. In H. Landrine & N. F. Russo (Eds.), Handbook of diversity in feminist psychology (pp. 83–121). New York: Springer.Google Scholar
- Costello, E. J., Angold, A., Burns, B. J., Erkanli, A., Stangl, D. K., & Tweed, D. L. (1996). The great smoky mountains study of youth: Functional impairment and serious emotional disturbance. Archives of General Psychiatry, 53, 1137–1143. https://doi.org/10.1001/archpsyc.1996.01830120077013.CrossRefPubMedGoogle Scholar
- Costello, E. J., He, J., Sampson, N. A., Kessler, R. C., & Merikangas, K. R. (2014). Services for adolescents with psychiatric disorders: 12-month data from the National Comorbidity Survey-Adolescent. Psychiatric Services, 65, 359–366. https://doi.org/10.1176/appi.ps.201100518.CrossRefPubMedGoogle Scholar
- Domino, M. E., Burns, B. J., Silva, S. G., Kratochvil, C. J., Vitiello, B., Reinecke, M. A.,… March, J. S. (2008). Cost-effectiveness of treatments for adolescent depression: Results from TADS. American Journal of Psychiatry, 165, 588–596. https://doi.org/10.1176/appi.ajp.2008.07101610.CrossRefPubMedGoogle Scholar
- Domino, M. E., Foster, E. M., Vitiello, B., Kratochvil, C. J., Burns, B. J., Silva, S. G.,… March, J. S. (2009). Relative cost-effectiveness of treatments for adolescent depression: 36-Week results from the TADS randomized trial. Journal of the American Academy of Child & Adolescent Psychiatry, 48, 711–720. https://doi.org/10.1097/CHI.0b013e3181a2b319.CrossRefGoogle Scholar
- Farmer, E. M. Z., Angold, A., Burns, B. J., & Costello, E. J. (1994). Reliability of self-reported service use: Test-retest consistency of children’s responses to the Child and Adolescent Services Assessment (CASA). Journal of Child and Family Studies, 3, 307–325. https://doi.org/10.1007/BF02234688.CrossRefGoogle Scholar
- Farmer, E. M. Z., Mustillo, S. A., Wagner, H. R., Burns, B. J., Kolko, D. J., Barth, R. P., & Leslie, L. K. (2010). Service use and multi-sector use for mental health problems by youth in contact with child welfare. Children and Youth Services Review, 32, 815–821. https://doi.org/10.1016/j.childyouth.2010.01.019.CrossRefGoogle Scholar
- Geisinger, K. F. (2013). Reliability. In K. F. Geisinger (Ed.), APA handbook of testing and assessment in psychology: Vol. 1. Test theory and testing and assessment in industrial and organizational psychology (pp. 21–42). Washington, DC: American Psychological Association.Google Scholar
- Ghandour, R. M., Kogan, M. D., Blumberg, S. J., Jones, J. R., & Perrin, J. M. (2012). Mental health conditions among school-aged children: Geographic and sociodemographic patterns in prevalence and treatment. Journal of Developmental & Behavioral Pediatrics, 33, 42–54. https://doi.org/10.1097/DBP.0b013e31823e18fd.CrossRefGoogle Scholar
- Guy, W. (1976). Clinical global impression scale. Rockville, MD: National Institute of Mental Health.Google Scholar
- Hansen, N. B., Lambert, M. J., & Forman, E. M. (2002). The psychotherapy dose-response effect and its implications for treatment delivery services. Clinical Psychology: Science and Practice, 9, 329–343.Google Scholar
- Hurlburt, M. S., Leslie, L. K., Landsverk, J., Barth, R. P., Burns, B. J., Gibbons, R. D.,… Zhang, J. (2004). Contextual predictors of mental health service use among children open to child welfare. Archives of General Psychiatry, 61, 1217–1224. https://doi.org/10.1001/archpsyc.61.12.1217.CrossRefPubMedPubMedCentralGoogle Scholar
- Hwang, W., Myers, H. F., Abe-Kim, J., & Ting, J. Y. (2008). A conceptual paradigm for understanding culture’s impact on mental health: The cultural influences on mental health (CIMH) model. Clinical Psychology Review, 28, 211–227. https://doi.org/10.1016/j.cpr.2007.05.001.CrossRefPubMedGoogle Scholar
- Jensen, P. S., Hinshaw, S. P., Swanson, J. M., Greenhill, L. L., Conners, C. K., Arnold, L. E.,… Wigal, T. (2001). Findings from the NIMH Multimodal Treatment Study of ADHD (MTA): Implications and applications for primary care providers. Developmental and Behavioral Pediatrics, 22, 60–73.CrossRefGoogle Scholar
- Kazdin, A. E., Holland, L., Crowley, M., & Breton, S. (1997). Barriers to treatment participation scale: Evaluation and validation in the context of child outpatient treatment. Journal of Child Psychology and Psychiatry, 38, 1051–1062. https://doi.org/10.1111/j.1469-7610.1997.tb01621.x.CrossRefPubMedGoogle Scholar
- Kutner, M. H., Nachtsheim, C. J., & Neter, J. (2004). Applied linear regression models (4th edn.). Irwin: McGraw-Hill.Google Scholar
- Lynch, F. L., Dickerson, J. F., Clarke, G., Vitiello, B., Porta, G., Wagner, K. D., … Brent, D. (2011). Incremental cost-effectiveness of combined therapy vs medication only for youth with Selective Serotonin Reuptake Inhibitor–resistant depression: Treatment of SSRI-Resistant Depression in Adolescents Trial findings. Archives of General Psychiatry, 68, 253–262.CrossRefGoogle Scholar
- Mayr, U., & Kliegl, R. (2000). Task-set switching and long-term memory retrieval. Journal of Experimental Psychology: Learning, Memory, and Cognition, 5, 1124–1140.Google Scholar
- Merikangas, K. R., He, J., Burstein, M. E., Swendsen, J., Avenevoli, S., Case, B.,… Olfson, M. (2011). Service utilization for lifetime mental disorders in U.S. adolescents: Results of the National Comorbidity Survey Adolescent supplement (NCS-A). Journal of the American Academy of Child & Adolescent Psychiatry, 50, 32–45. https://doi.org/10.1016/j.jaac.2010.10.006.CrossRefGoogle Scholar
- Sanchez, A. L., Cornacchio, D., Poznanski, B., Golik, A. M., Chou, T., & Comer, J. S. (2018). The effectiveness of school-based mental health services for elementary-aged children: A meta-analysis. Journal of the American Academy of Child & Adolescent Psychiatry, 57, 153–165. https://doi.org/10.1016/j.jaac.2017.11.022.CrossRefGoogle Scholar
- Spielberger, C. D., Gorsuch, R. L., Lushene, R., Vagg, P. R., & Jacobs, G. A. (1983). Manual for the state-trait anxiety inventory. Palo Alto, CA: Consulting Psychologists Press.Google Scholar
- U.S. Department of Health and Human Services. (2001). Mental health: Culture, race, and ethnicity—a supplement to mental health: A report of the surgeon general. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services.CrossRefGoogle Scholar
- Walker, L. S., & Garber, J. (2003). Manual for the children’s somatization inventory. Nashville, TN: Vanderbilt University Medical Center.Google Scholar
- Weersing, V. R., Brent, D. A., Rozenman, M. S., Gonzalez, A., Jeffreys, M., Dickerson, J. F.,… Iyengar, S. (2017). Brief behavioral therapy for pediatric anxiety and depression in primary care: A randomized clinical trial. JAMA Psychiatry. https://doi.org/10.1001/jamapsychiatry.2017.0429.CrossRefPubMedPubMedCentralGoogle Scholar