Multilevel Confirmatory Factor Analysis of the Systems of Care Implementation Survey (SOCIS)

  • Paul E. Greenbaum
  • Wei Wang
  • Roger Boothroyd
  • Krista Kutash
  • Robert M. Friedman
Article
First Online:

Abstract

A major impediment to obtaining national information on systems of care implementation has been the lack of a psychometrically sound large-scale survey instrument. The present study provided information on the factorial and concurrent validity of the Systems of Care Implementation Survey scales. Multilevel confirmatory factor analysis and multilevel regression analysis were used to test these indicators of internal and external validity. Two hundred twenty-five counties were randomly selected and stratified by population size and poverty level. Nine hundred ten informants responded to the survey questionnaire, M = 4.04 informants per county (SD = 3.17). Results indicated that all models had at least adequate fit to the data, with nine of the 14 factor models having excellent fit. Overall, 11 of the 14 factors had some indication that receiving federal funding to create systems of care was associated with higher scores on the factors. Implications for future research were discussed.

Keywords

Transformational Leadership Performance Measurement System Standardize Loading Multilevel Regression Analysis Care Implementation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

This research was partially supported by Grant H133B90004 from the Center for Mental Health Services, Substance Abuse and Mental Health Administration and the National Institute for Disability and Rehabilitation Research. The opinions contained in this manuscript are those of the authors and do not necessarily reflect those of either the US Department of Education or the Center for Mental Health Services, SAMSHA. We gratefully acknowledge helpful comments on the statistical analyses by members of the Prevention Science and Methodology Group (PI: C. Hendricks Brown) supported by the National Institute of Mental Health and the National Institute on Drug Abuse through Grant 5R01MH40859.

References

  1. 1.
    Stroul BA, Friedman RM. A System of Care for Severely Emotionally Disturbed Children and Youth. Washington, DC: Georgetown University, 1986.Google Scholar
  2. 2.
    Boothroyd RA, Greenbaum PE, Wang W, et al. Development of a measure to assess the implementation of children’s systems of care: The System of Care Implementation Survey (SOCIS). Journal of Behavioral Health Services and Research, 2011; 38(3).Google Scholar
  3. 3.
    Procter EK, Landsverk J, Aarons G, et al. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Administration and Policy in Mental Health, 2009; 36:24–34.CrossRefGoogle Scholar
  4. 4.
    Raudenbush SW, Rowan B, Kang, SJ. A multilevel, multivariate model for studying school climate with estimation via the EM algorithm and application to U.S. high-school data. Journal of Educational Statistics, 1991; 16:295–330.CrossRefGoogle Scholar
  5. 5.
    Muthén BO. Multilevel factor analysis of class and student achievement components. Journal of Educational Measurement, 1990; 28:338–354.CrossRefGoogle Scholar
  6. 6.
    Muthén BO. Multilevel covariance structure analysis. Sociological Methods and Research, 1994; 22:376–398.CrossRefGoogle Scholar
  7. 7.
    Berman E, Wang X. Performance measurement in U. S. counties: capacity for reform. Public Administration Review. 2000: 60, 409–420.CrossRefGoogle Scholar
  8. 8.
    Health Resources and Services Administration. Community health status report. Washington, D.C.; Health Resources and Services Administration: 2000.Google Scholar
  9. 9.
    Andrews FM. Construct validity and error components of survey measures: a structural modeling approach. Public Opinion Quarterly, 1984; 48:409–442.CrossRefGoogle Scholar
  10. 10.
    Zypher MJ, Kaplan SA, & Christian MS. Assumptions of cross-level measurement and structural invariance in the analysis of multilevel data: problems and solutions. Group Dynamics: Theory, Research, and Practice, 2008; 12:127–140.CrossRefGoogle Scholar
  11. 11.
    Muthén LK, Muthén BO. Mplus User’s Guide. Fifth Edition. Los Angeles, CA: Muthén & Muthén, 1998–2007.Google Scholar
  12. 12.
    Schafer JL, Graham JW. Missing data: our view of the state of the art. Psychological Methods, 2001; 7:147–177.CrossRefGoogle Scholar
  13. 13.
    Bentler PM. On the fit of models to covariances and methodology to the Bulletin. Psychological Bulletin, 1992; 112:400-404.PubMedCrossRefGoogle Scholar
  14. 14.
    Tucker LR, Lewis C. The reliability coefficient for maximum likelihood factor analysis. Psychometrika, 1973; 38:1–10.CrossRefGoogle Scholar
  15. 15.
    Steiger JH, Lind JC. Statistically-based Tests for the Number of Common Factors. Paper presented at the annual meeting of the Psychonomic Society, Iowa City, IA, May 1980.Google Scholar
  16. 16.
    McDonald RP, Ho M. Principles and practice in reporting structural equation analyses. Psychological Methods. 2002; 7: 64–82.PubMedCrossRefGoogle Scholar
  17. 17.
    Cicchetti DV, Sparrow SS. Developing criteria for establishing the interrater reliability of specific items in a given inventory. American Journal of Mental Deficiencies. 1981; 86:127–137.Google Scholar
  18. 18.
    Hox J. Multilevel Analysis: Techniques and Applications. Mahwah, NJ: Erlbaum, 2002.Google Scholar
  19. 19.
    Kamata A, Bauer DJ, Miyazaki Y. Multilevel measurement modeling. In AA O’Connell and DB McCoach (eds). Multilevel Modeling of Educational Data. Charlotte, NC: IAP, 2008: 345–388.Google Scholar
  20. 20.
    Raudenbush SW, Bryk AS. Hierarchical Linear Models: Applications and Data Analysis Methods. 2 nd Edition. Thousand Oaks, CA: Sage, 2002.Google Scholar
  21. 21.
    Fleiss JL. Statistical Methods for Rates and Proportions. 2 nd Edition. New York: Wiley, 1981.Google Scholar
  22. 22.
    Kutash K, Greenbaum P, Wang W, et al. Levels of system of care implementation: A national benchmarking study. Journal of Behavioral Health Services and Research, 2011; 38(3).Google Scholar
  23. 23.
    Lunn LM, Heflinger CA, Wang W, et al. Community characteristics and implementation factors associated with effective systems of care. Journal of Behavioral Health Services and Research, 2011; 38(3).Google Scholar

Copyright information

© National Council for Community Behavioral Healthcare 2011

Authors and Affiliations

  • Paul E. Greenbaum
    • 1
  • Wei Wang
    • 1
  • Roger Boothroyd
    • 1
  • Krista Kutash
    • 1
  • Robert M. Friedman
    • 1
  1. 1.Department of Child and Family StudiesUniversity of South FloridaTampaUSA

Personalised recommendations