Abstract
Tests that measure the emotional and behavioral problems of children and youth are typically not normed and standardized on youth diagnosed with disruptive behavior, particularly those youth in residential care. Yet professional standards mandate that before instruments are used with a specific population the psychometric properties need to be studied and re-established: specifically, psychometric properties, including validity, need to be evaluated (AERA, APA, and NCME, The standards for educational and psychological testing. AERA, Washington, DC, 1999). The purpose of the present study was to assess the validity characteristics of the Symptoms and Functioning Severity Scale (SFSS; Bickman et al., Manual of the Peabody Treatment Progress Battery, Vanderbilt University, Nashville, TN, 2010), a widely used test developed for use in outpatient clinics, with youth in a residential care program. The convergent validity of the SFSS was established with the large correlations (0.78–0.86) with the CBCL. Several binary classification analyses including specificity, area under the receiver operating characteristic curve, positive and negative likelihood ratios, and the Youden Index supported the validity of the SFSS. However, the sensitivity index was somewhat low indicating the test may produce a high level of false negatives. Limitations, future research and implications are discussed.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Achenbach, T. M., & Rescorla, L. (2001). ASEBA school-age forms & profiles. Burlington, VT: Research Center for Children, Youth, and Families.
American Educational Research Association (AERA), American Psychological Association (APA), and National Council of Measurement in Education (NCME). (1999). The standards for educational and psychological testing. Washington, DC: AERA.
Arseneault, L., Kim-Cohen, J., Taylor, A., Caspi, A., & Moffitt, T. E. (2005). Psychometric evaluation of 5-and 7-year-old children’s self-reports of conduct problems. Journal of Abnormal Child Psychology, 33, 537–550.
Athay, M. M., Riemer, M., & Bickman, L. (2012). The Symptoms and Functioning Severity Scale (SFSS): Psychometric evaluation and discrepancies among youth, caregiver, and clinician ratings over time. Administration and Policy in Mental Health and Mental Health Services Research, 39, 13–29.
Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Association of Child and Adolescent Psychiatry, 47, 1114–1119.
Bickman, L. (2012). Why can’t health services be more like baseball? Administration and Policy in Mental Health and Mental Health Services Research, 39(1), 1–2.
Bickman, L., Athay, M. M., Riemer, M., Lambert, E. W., Kelley, S. D., et al. (Eds.). (2010). Manual of the Peabody Treatment Progress Battery (2nd ed. [Electronic version]). Nashville, TN: Vanderbilt University. http://peabody.vanderbilt.edu/ptpb/. Accessed 30 Aug 2010.
Bickman, L., Kelley, S. D., Breda, C., de Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomized trial. Psychiatric Services, 62(11), 1423–1429.
Bradley, R., Doolittle, J., & Bartolotta, R. (2008). Building on the data and adding to the discussion: The experiences and outcomes of students with emotional disturbance. Journal of Behavioral Education, 17, 4–23.
Brown, J. D., Hamilton, M., Natzke, B., Ireys, H. T., & Gillingham, M. (2011). Use of out-of-home care among a statewide population of children and youth enrolled in medicaid. Journal of Child and Family Studies, 20, 48–56.
Charles, J. M., Bywater, T., & Edwards, R. T. (2011). Parenting interventions: A systematic review of the economic evidence. Child: Care Health and Development, 37, 462–474.
Chorpita, B. F., Reise, S., Weisz, J. R., Grubbs, K., Becker, K. D., & Krull, J. L. (2010). Evaluation of the brief problem checklist: Child and caregiver interviews to measure clinical progress. Journal of Consulting and Clinical Psychology, 78(4), 526–536.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Earlbaum.
Compton, D. L., Fuchs, D., Fuchs, L. S., & Bryant, J. D. (2006). Selecting at-risk readers in first grade for early intervention: A two-year longitudinal study of decision rules and procedures. Journal of Educational Psychology, 98, 394–409.
Cook, N. R. (2007). Use and misuse of the receiver operating characteristic curve in risk prediction. Ciculation, 115, 928–935.
Cuthbert, R., Pierre, J. S., Stewart, S. L., Cook, S., Johnson, A. M., & Leschied, A. W. (2011). Symptom persistence in seriously emotionally disordered children: Findings of a two-year follow-up after residental treatment. Child & Youth Care Forum, 40, 267–280.
Davis, J. L., & Daly, D. L. (2003). Girls and Boys Town long-term residential program training manual (4th ed.). Boys Town, NE: Father Flanagan’s Boys’ Home.
Duppong Hurley, K., Lambert, M. C., & Stevens, A. (2014). Psychometrics of the symptoms, functioning, and severity scale. Journal of Emotional and Behavioral Disorders. doi:10.1177/1063426614535809.
Duppong Hurley, K., Lambert, M. C., Van Ryzin, M., Sullivan, J., & Stevens, A. (2013). Therapeutic alliance between youth and staff in residential group care: Psychometrics of the therapeutic alliance quality scale. Children and Youth Services Review, 35(1), 56–64.
Duppong Hurley, K., Stevens, A., Lambert, M. C., & Sullivan, J. (2012). Bringing it all together: The relationships among implementation, therapeutic factors, and youth outcomes. A paper session presented at the annual conference for the American Evaluation Association, Minneapolis, MN.
Ebesutani, C., Berstein, A., Chorpita, B., & Weisz, J. R. (2012). A transportable assessment protocol for prescribing youth psychological treatments in real-world settings: Reducing assessment burden via self-report scales. Psychological Assessment, 24(1), 141–155.
Epstein, M. H. (2002). Behavioral and Emotional Rating Scale, second edition examiner’s manual. Austin, TX: Pro-Ed.
Goodman, R. (1997). The strengths and difficulties questionnaire: A research note. Journal of Child Psychology and Psychiatry, 38, 581–586.
Johnson, E. S., Jenkins, J. R., Petscher, Y., & Catts, H. W., (2009). How can we improve the accuracy of screening instruments? Learning Disabilities Research & Practice, 24, 174–185. doi:10.1111/j.1540-5826.2009.00291.x.
Kazdin, A. E. (1994). Methodology, design and evaluation in psychotherapy research. In A. E. Bergin, & S. L. Garfield (Eds.), Handbook of psychotherapy and behavioral change (4th ed., pp. 19–71). New York, NY: Wiley.
Kolko, D. J., & Pardini, D. A. (2010). ODD dimensions, ADHD, and callous–unemotional traits as predictors of treatment response in children with disruptive behavior disorders. Journal of Abnormal Psychology, 119, 713.
McKinney, C., & Morse, M. (2012). Assessment of disruptive behavior disorders: Tools and recommendations. Professional Psychology: Research and Practice, 43, 641.
Park, S. H., Goo, J. M., & Jo, C. H. (2004). Receiver operating characteristic (ROC) curve: Practical review for radiologists. Korean Journal of Radiology, 5(1), 11–18.
Reynolds, C. R., & Kamphaus, R. W. (2004). Behavior assessment system for children – second edition manual. Minneapolis, MN: Pearson.
Shaffer, D., Fisher, P. W., Lucas, C. P., Dulcan, M., & Schwab-Stone, M. E. (2000). NIMH Diagnostic Interview Schedule for Children, version IV (NIMH DISC-IV): Description, differences from previous versions, and reliability of some common diagnoses. Journal of American Academy of Child and Adolescent Psychiatry, 39, 28–38.
Short, R. J., & Shapiro, S. K. (1993). Conduct disorders: A framework for understanding and intervention in schools and communities. School Psychology Review, 22, 362–375.
Trentacosta, C. J., Hyde, L. W., Goodlett, B. D., & Shaw, D. S. (2013). Longitudinal prediction of disruptive behavior disorders in adolescent males from multiple risk domains. Child Psychiatry and Human Development, 44, 561–572.
Webster-Stratton, C. (1993). Strategies for helping early school-aged children with oppositional defiant and conduct disorders: The importance of home-school partnerships. School Psychology Review, 22, 437–457.
Wolf, M. M., Phillips, E., Fixsen, D., Baukmann, C. J., Kirigin, K. A., & Willner, A. G. (1976). Achievement Place: The teaching-family model. Child Care Quarterly, 5, 92–103.
Wood, F., Flowers, L., Meyer, M., & Hill, D. (2002, November). How to evaluate and compare screening tests: Principles of science and good sense. Paper presented at the meeting of the International Dyslexia Association, Atlanta, GA.
Youden, W. J. (1950). Index for diagnostic tests. Cancer, 3(1), 32–35.
Acknowledgments
The research reported herein was supported, in part, by the National Institute of Mental Health through Grant R34MH080941 and by the Institute of Education Sciences, U.S. Department of Education, through Grant R324B110001 to the University of Nebraska-Lincoln. The opinions expressed are those of the authors and do not represent views of the National Institute of Mental Health or the U.S. Department of Education.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lambert, M.C., Hurley, K.D., Gross, T.J. et al. Validation of the Symptoms and Functioning Severity Scale in Residential Group Care. Adm Policy Ment Health 42, 356–362 (2015). https://doi.org/10.1007/s10488-014-0575-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-014-0575-z