Advertisement

Reliability and Validity of the Thinking Skills Inventory, a Screening Tool for Cross-Diagnostic Skill Deficits Underlying Youth Behavioral Challenges

  • Lu Wang
  • Alisha R. PollastriEmail author
  • Pieter J. Vuijk
  • Erin N. Hill
  • Brenda A. Lee
  • Anna Samkavitz
  • Ellen B. Braaten
  • J.  Stuart Ablon
  • Alysa E. Doyle
Article

Abstract

Deficits in a range of skill domains (including executive functioning, emotion regulation, social cognition and language/communication) are associated with disrupted youth behavior and functioning across mental health diagnoses. The identification of skill deficits are important for effective treatment planning, particularly for personalized interventions. While there are multiple ways to assess these skills, parent/caregiver reports represent an important information source. To date, no single, brief measure has been developed that gathers parent/caregiver ratings across this range of constructs. We have developed a short caregiver-report questionnaire (the Thinking Skills Inventory; TSI), to screen for skill deficits. Here, we examine the reliability and validity of this rating scale in 384 youth who were consecutively referred for neuropsychiatric evaluation. A primary caregiver completed the TSI as well as other established measures. Exploratory and confirmatory factor analyses support five subscales on the TSI: Attention and Working Memory, Language and Communication, Emotion Regulation, Cognitive Flexibility, and Social Thinking Skills. The subscales showed moderate to high internal consistency (Cronbach’s alphas range from 0.84 to 0.91). Correlations with established caregiver-report measures confirm their convergent and discriminant validity, and associations with multiple clinical diagnoses and cross-diagnostic aggressive behavior further support the utility of the scale for our intended purpose. In sum, this free, brief measure is a valid and reliable way to identify variation in skill domains relevant to a range of psychopathology. The TSI may be useful in youth mental health settings to assist with treatment planning and to inform referral for further evaluation.

Keywords

Cross-diagnostic traits Child psychopathology Assessment Collaborative problem solving 

Notes

Funding

This research was supported in part by the David Judah Fund, the Stanley Center for Psychiatric Research, and NIH R03MH106862 awarded to Dr. Alysa E. Doyle.

Compliance with Ethical Standards

Conflict of Interest

Drs. Pollastri, Wang, Hill, and Ablon work for Think:Kids, a non-profit program in the Department of Psychiatry at Massachusetts General Hospital, from which staff train and consult on Collaborative Problem Solving and use the Thinking Skills Inventory (TSI). The TSI is available at no cost to paid clients of Think:Kids as well as to the larger research and clinical communities.

Experiment Participants

This research was approved by the appropriate Institutional Review Board and all participants provided informed consent.

Supplementary material

10862_2018_9703_MOESM1_ESM.docx (14 kb)
Supplementary Table 1 (DOCX 14 kb)
10862_2018_9703_MOESM2_ESM.docx (15 kb)
Supplementary Table 2 (DOCX 15 kb)

References

  1. Achenbach, T. M. (2009). ASEBA: Development, findings, theory and applications. Burlington, VT: University of Vermont Research Center for Children, Youth and Families.Google Scholar
  2. Achenbach, T. M., & Rescorla, L. A. (2001). Manual for the ASEBA school-age forms and profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, and Families ISBN 0-938565-73-7.Google Scholar
  3. Allen, M. J., & Yen, W. M. (2002). Introduction to measurement theory. Long Grove: Waveland Press.Google Scholar
  4. American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., Text Revision). Washington, DC: Author.Google Scholar
  5. American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: Author.Google Scholar
  6. Bentler, P. M., & Bonett, D. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), 588–606.  https://doi.org/10.1037/0033-2909.88.3.588.CrossRefGoogle Scholar
  7. Berkovits, L. D., & Baker, B. L. (2014). Emotion dysregulation and social competence: Stability, change and predictive power. Journal of Intellectual Disability Research, 58(8), 765–776.  https://doi.org/10.1111/jir.12088.PubMedCrossRefGoogle Scholar
  8. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & S. L. Long (Eds.), Testing structural equation models (pp. 136–162). London: Sage Publications.Google Scholar
  9. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Routledge.Google Scholar
  10. Dodge, K. A., Lansford, J. E., Burks, V. S., Bates, J. E., Pettit, G. S., Fontaine, R., & Price, J. M. (2003). Peer rejection and social information-processing factors in the development of aggressive behavior problems in children. Child Development, 74(2), 374–393.PubMedPubMedCentralCrossRefGoogle Scholar
  11. Dolan, C. (1994). Factor analysis of variables with 2, 3, 5, and 7 response categories: A comparison of categorical variable estimators using simulated data. British Journal of Mathematical and Statistical Psychology, 47(2), 309–326.CrossRefGoogle Scholar
  12. Doyle, A. E., Vuijk, P. J., Doty, N. D., McGrath, L. M., Willoughby, B. L., O’Donnell, E. H., et al. (2017). Cross-disorder cognitive impairments in youth referred for neuropsychiatric evaluation. Journal of the International Neuropsychological Society, 1–13.Google Scholar
  13. Eisenberg, N., Fabes, R. A., Bernzweig, J., Karbon, M., Poulin, R., & Hanish, L. (1993). The relations of emotionality and regulation to preschoolers’ social skills and sociometric status. Child Development, 64(5), 1418–1438.  https://doi.org/10.2307/1131543.PubMedCrossRefGoogle Scholar
  14. Gioia, G. A. (2000). BRIEF: Behavior rating inventory of executive function: Professional manual. Odessa: Psychological Assessment Resources.Google Scholar
  15. Gioia, G., Isquith, P. K., Guy, S. C., Kenworthy, L., & Reviewed by Baron, I.S. (2000). Test review: Behavior rating inventory of executive function. Child Neuropsychology, 6(3), 235–238.CrossRefGoogle Scholar
  16. Gioia, G. A., Isquith, P. K., Guy, S. C., & Kenworthy, L. (2015). Behavior rating inventory of executive functions-2. Lutz: PAR.Google Scholar
  17. Greene, R. W. (1998). The explosive child: A new approach for understanding and parenting easily frustrated, chronically inflexible children. New York: Harper Collins.Google Scholar
  18. Greene, R. W., & Ablon, J. S. (2005). Treating explosive kids: The collaborative problem-solving approach. New York: Guilford Press.Google Scholar
  19. Harrison, P.L. & Oakland, T. (2003). Adaptive Behavior Assessment System – Second Edition, San Antonio: The Psychological Corporation.Google Scholar
  20. Hendricks, M. A., & Buchanan, T. W. (2016). Individual differences in cognitive control processes and their relationship to emotion regulation. Cognition & Emotion, 30(5), 912–924.  https://doi.org/10.1080/02699931.2015.1032893.CrossRefGoogle Scholar
  21. Hill, A. L., Degnan, K. A., Calkins, S. D., & Keane, S. P. (2006). Profiles of externalizing behavior problems for boys and girls across preschool: The roles of emotion regulation and inattention. Developmental Psychology, 42(5), 913–928.PubMedCrossRefGoogle Scholar
  22. Hooper, D., Coughlan, J., & Mullen, M. (2008). Structural equation modeling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, 6(1), 53–60.Google Scholar
  23. Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453.CrossRefGoogle Scholar
  24. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55.CrossRefGoogle Scholar
  25. Jamshidian, M., Jalal, S., & Jansen, C. (2014). MissMech: An R package for testing homoscedasticity, multivariate normality, and missing completely at random (MCAR). Journal of Statistical Software, 56(6), 1–31 Available from http://www.jstatsoft.org/v56/i06/.CrossRefGoogle Scholar
  26. Kline, P. (2013). Handbook of psychological testing. London, England. Routledge.Google Scholar
  27. Korkmaz, S., Goksuluk, D., & Zararsiz, G. (2014). MVN: An R package for assessing multivariate normality. The R Journal, 6(2), 151–162.CrossRefGoogle Scholar
  28. Kotov, R., Ruggero, C. J., Krueger, R. F., Watson, D., Yuan, Q., & Zimmerman, M. (2011). New dimensions in the quantitative classification of mental illness. Archives of General Psychiatry, 68, 1003–1011.PubMedCrossRefGoogle Scholar
  29. Lindell, M. K., & Whitney, D. J. (2001). Accounting for common method variance in cross-sectional research designs. Journal of Applied Psychology, 86(1), 114–121.  https://doi.org/10.1037//0021-9010.86.1.114.PubMedCrossRefGoogle Scholar
  30. Lopes, P. N., Salovey, P., Côté, S., & Beers, M. (2005). Emotion regulation abilities and the quality of social interaction. Emotion, 5(1), 113–118.  https://doi.org/10.1037/1528-3542.5.1.113.PubMedCrossRefGoogle Scholar
  31. MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130–149.CrossRefGoogle Scholar
  32. Matthys, W., Cuperus, J. M., & Engeland, H. V. (1999). Deficient social problem-solving in boys with ODD/CD, with ADHD, and with both disorders. Journal of the American Academy of Child & Adolescent Psychiatry, 38(3), 311–321.CrossRefGoogle Scholar
  33. Meyers, E., DeSerisy, M., & Roy, A. K. (2017). Disruptive mood dysregulation disorder (DMDD): An RDoC perspective. Journal of Affective Disorders, 216, 117–122.  https://doi.org/10.1016/j.jad.2016.08.007.PubMedCrossRefGoogle Scholar
  34. Morris, S. E., & Cuthbert, B. N. (2012). Research domain criteria: Cognitive systems, neural circuits, and dimensions of behavior. Dialogues in Clinical Neuroscience, 14(1), 29–37.PubMedPubMedCentralGoogle Scholar
  35. Nordhausen, K., & Oja, H. (2011). Multivariate L1 methods: The package MNM. Journal of Statistical Software, 43, 1–28.CrossRefGoogle Scholar
  36. Nunnally, J. C. (1978). Psychometric theory. New York: McGraw-Hill.Google Scholar
  37. Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903.  https://doi.org/10.1037/0021-9010.88.5.879.PubMedCrossRefGoogle Scholar
  38. Pollastri, A. R., Epstein, L. D., Heath, G. H., & Ablon, J. S. (2013). The collaborative problem solving approach: Outcomes across settings. Harvard Review of Psychiatry, 21(4), 188–199.PubMedGoogle Scholar
  39. Renna, M. E., Quintero, J. M., Fresco, D. M., & Mennin, D. S. (2017). Emotion regulation therapy: A mechanism-targeted treatment for disorders of distress. Frontiers in Psychology, 8, 1–14.CrossRefGoogle Scholar
  40. Revelle, W. (2017). psych: Procedures for Personality and Psychological Research, Northwestern University, Evanston, Illinois, USA, https://CRAN.R-project.org/package=psychVersion = 1.7.8.
  41. Röll, J., Koglin, U., & Petermann, F. (2012). Emotion regulation and childhood aggression: Longitudinal associations. Child Psychiatry & Human Development, 43(6), 909–923.CrossRefGoogle Scholar
  42. Rosseel, Y. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36.CrossRefGoogle Scholar
  43. RStudio Team. (2016). RStudio: integrated development for R. Boston, MA: RStudio, Inc. Available from http://www.rstudio.com/.
  44. Rushton, J., Bruckman, D., & Kelleher, K. (2002). Primary care referral of children with psychosocial problems. Archives of Pediatrics & Adolescent Medicine, 156(6), 592–598.CrossRefGoogle Scholar
  45. Russell, D. (2002). In search of underlying dimensions: The use (and abuse) of factor analysis in personality and social psychology bulletin. Personality and Social Psychology Bulletin, 28(12), 1629–1646.Google Scholar
  46. Steiger, J. H. (2007). Understanding the limitations of global fit assessment in structural equation modeling. Personality and Individual Differences, 42(5), 893–898.CrossRefGoogle Scholar
  47. Stieben, J., Lewis, M. D., Granic, I., Zelazo, P. D., Segalowitz, S., & Pepler, D. (2007). Neurophysiological mechanisms of emotion regulation for subtypes of externalizing children. Development and Psychopathology, 19(2), 455–480.PubMedCrossRefGoogle Scholar
  48. Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Allyn & Bacon/Pearson Education.Google Scholar
  49. Van Goozen, S. H., Cohen-Kettenis, P. T., Snoek, H., Matthys, W., Swaab-Barneveld, H., & Van Engeland, H. (2004). Executive functioning in children: A comparison of hospitalized ODD and ODD/ADHD children and normal controls. Journal of Child Psychology and Psychiatry, 45(2), 284–292.PubMedCrossRefGoogle Scholar
  50. Zadeh, Z. Y., Im-Bolter, N., & Cohen, N. J. (2007). Social cognition and externalizing psychopathology: An investigation of the mediating role of language. Journal of Abnormal Child Psychology, 35(2), 141–152.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Lu Wang
    • 1
    • 2
  • Alisha R. Pollastri
    • 1
    • 2
    Email author
  • Pieter J. Vuijk
    • 1
    • 2
    • 3
    • 4
  • Erin N. Hill
    • 1
    • 2
  • Brenda A. Lee
    • 1
    • 2
    • 3
    • 4
  • Anna Samkavitz
    • 1
    • 2
    • 3
    • 4
  • Ellen B. Braaten
    • 1
    • 2
  • J.  Stuart Ablon
    • 1
    • 2
  • Alysa E. Doyle
    • 1
    • 2
    • 3
    • 4
  1. 1.Department of PsychiatryMassachusetts General HospitalBostonUSA
  2. 2.Harvard Medical SchoolBostonUSA
  3. 3.Center for Genomic MedicineMassachusetts General HospitalBostonUSA
  4. 4.The Stanley Center for Psychiatric Research, Broad InstituteCambridgeUSA

Personalised recommendations