Skip to main content

Advertisement

SpringerLink
Log in

Development of a Measure to Assess the Implementation of Children’s Systems of Care: The Systems of Care Implementation Survey (SOCIS)

  • Published:
The Journal of Behavioral Health Services & Research Aims and scope Submit manuscript

Abstract

The children’s system of care framework has been extensively implemented in the U.S. Since its inception in 1993, the Comprehensive Community Mental Health Services for Children and Their Families Program has invested in excess of $1 billion supporting the development of systems of care in 164 grantee sites across the country. Despite these efforts to implement children’s systems of care nationally, little is known about the extent to which the principles and values actually have been put into practice outside of the funded grantee sites. This paper describes the development of the Systems of Care Implementation Survey, a measure designed specifically for the first ever study assessing the level of implementation of factors contributing to effective children’s systems of care in a nationally representative sample of counties throughout the U.S.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. President’s Commission on Mental Health. Report of the Sub-task Panel on Infants, Children and Adolescents, Washington, D.C. 1978.

  2. US Congress. Children’s Mental Health: Problems and Services—a Background Paper, Washington, D.C.: Office of Technology Assessment; 1986

    Google Scholar 

  3. President’s New Freedom Commission on Mental Health. Achieving the promise: transforming mental health care in America, Rockville, MD: Department of Health and Human Services; 2003.

    Google Scholar 

  4. US Public Health Service Report of the Surgeon General’s Conference on Children’s Mental Health. A National Action Agenda, Washington, D.C.: Department of Health and Human Services; 2000

    Google Scholar 

  5. Friesen BJ, Huff B. Family perspectives on systems of care. In: Stroul BA, ed. Children’s Mental Health: Creating Systems of Care in a Changing Society. Baltimore, MD: Paul H. Brookes Publishing Co., Inc; 1996:41–68.

    Google Scholar 

  6. Isaacs-Shockley M, Cross T, Bazron B, et al. Framework for a culturally competent system of care. In: Stroul BA, ed. Children’s mental health: creating system of care in a changing society. Baltimore, MD: Paul H. Brookes Publishing Co, Inc.; 1996:23–40

    Google Scholar 

  7. US Department of Health and Human Services, Office of Minority Health. (2001). Assuring Cultural Competence in Health Care: Recommendations for National Standards and an Outcomes-focused Research Agenda, Rockville, MD: IQ Solutions, Inc. Available at: http://www.omhrc.gov/clas/index.htm. Accessed on 4 Jan 2005.

  8. Stroul BA, Pires SA, Armstrong MI. Health Care Reform Tracking Project: tracking state health care reforms as they affect children and adolescents with behavioral health disorders and their families—2000 state survey. Tampa, FL: Research and Training Center for Children's Mental Health, Department of Child and Family Studies, Division of State and Local Support, Louis de la Parte Florida Mental Health Institute, University of South Florida; 2001

    Google Scholar 

  9. Stroul BA, Friedman RM. A system of care for severely emotionally disturbed children and youth, Washington, D. C.: Georgetown University, CASSP Technical Assistance Center; 1986.

    Google Scholar 

  10. Center for Mental Health Services (CMHS). Annual report to Congress on the evaluation of the comprehensive community mental health services for children and their families program, executive summary, 2004. Atlanta, GA: ORC Macro; 2004

    Google Scholar 

  11. Blau, GM, Huang, LN, Mallery, CJ. Advancing efforts to improve children’s mental health in America: a commentary. Administration and Policy in Mental Health and Mental Health Services Research, 2010; 37:140–144.

    Article  PubMed  Google Scholar 

  12. Stroul BA. Issue brief. Systems of care: a framework for system reform in children’s mental health, Washington, D.C.: National Technical Assistance Center for Children’s Mental Health, Georgetown University Center for Child and Human Development; 2002

    Google Scholar 

  13. Friedman RM, Hernandez M. The national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program: a commentary. Children’s Services: Social Policy, Research, and Practice 5(1): 2002; 67–74.

    Google Scholar 

  14. Hernandez M, Hodges S. Building upon the theory of change for systems of care. Journal of Emotional and Behavioral Disorders, 2003; 11:19–26.

    Article  Google Scholar 

  15. Stroul BA, Friedman R. The system of care concept and philosophy. In: Stroul, BA, ed. Children’s mental health: creating systems of care in a changing society. Baltimore, MD: Paul H. Brookes Publishing Co, Inc; 1996; 591–612

    Google Scholar 

  16. Holden EW, Friedman RM, Santiago RL. Overview of the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program. Journal of Emotional and Behavioral Disorders, 2001; 9:4–12.

    Article  Google Scholar 

  17. Center for Mental Health Services Annual Report to Congress on the Evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program, 2001. Atlanta, GA: ORC Macro, 2001.

  18. Vinson NB, Brannan AM, Baughman LN, et al. The system-of-care model: implementation in twenty-seven communities. Journal of Emotional and Behavioral Disorder. 2001; 9:30–42.

    Article  Google Scholar 

  19. Paulson R, Fixsen D, Friedman R. An analysis of implementation of systems of care at fourteen CMHS grant communities. Tampa, FL: Louis de la Parte Florida Mental Health Institute, University of South Florida, 2004

    Google Scholar 

  20. Clark LA, Watson D. Constructing validity: basic issues in objective scale development. Psychological Assessment. 1995; 7:309–319.

    Article  Google Scholar 

  21. Friedman RM. A model for implementing effective systems of care. In: C Newman, C Liberton K Kutash, RM Friedman eds. The 18th Annual Research Conference Proceedings, A System of Care for Children's Mental Health: Expanding the Research Base. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children’s Mental Health. 2007; 3–9.

    Google Scholar 

  22. Brannan AM, Baughman LN, Reed ED. System-of-care assessment: cross-site comparison of findings. Children's Services, Social Policy, Research, and Practice. 2002; 5:37–56.

    Google Scholar 

  23. Friedman RM, Fixsen D, Paulson R. Implementing effective systems of care: How can we improve? Paper presented at the 11th Annual Building on Family Strengths Conference, Portland, OR; 2004, May.

  24. Holden EW, De Carolis G, Huff B. Policy implications of the national evaluation of the Comprehensive Community Mental Health Services for Children and Their Families Program. Children's Services, Social Policy, Research, and Practice. 2002; 5:57–66.

    Google Scholar 

  25. Meridian Consulting Services, Inc. Integrated systems of care for children’s mental health: a technical assistance resource book. Albany, NY: Author; 1999.

  26. Pires SA. Primer hands on: the skill building curriculum for system of care leaders. Washington, DC: Georgetown University, National Technical Assistance Center for Children’s Mental Health, Center for Child and Human Development; 2003

    Google Scholar 

  27. Pumariega AJ, Winters NC, Huffine C. The evolution of systems of care for children’s mental health: forty years of community child and adolescent psychiatry, Community Mental Health Journal. 2003; 39:399–425.

    Article  PubMed  Google Scholar 

  28. Rosenblatt A, Woodbridge M. Deconstructing research on systems of care for youth with EBD: frameworks for policy research, Journal of Emotional and Behavioral Disorders. 2003; 11:27–38.

    Article  Google Scholar 

  29. Stephens RL, Holden EW, Hernandez M. System-of-care practice review scores as predictors of behavioral symptomatology and functional impairment. Journal of Child and Family Studies. 2004; 13:179–191.

    Article  Google Scholar 

  30. Gray B, Duran A, Segal A. eds. Revisiting the critical elements of Comprehensive Community Initiatives. Washington, DC: US Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation; 1997

    Google Scholar 

  31. Kubisch AC, Auspos P, Brown P, et al. Voices from the field II: Reflections on comprehensive community change. Queenstown, MD: Aspen Institute; 2002.

    Google Scholar 

  32. Bond LA, Hauf AMC. Taking stock and putting stock in primary prevention: characteristics of effective programs, The Journal of Primary Prevention. 2004; 24:199–221.

    Article  Google Scholar 

  33. Nation M, Crusto C, Wandersman A, et al. What works in prevention: principles of effective prevention programs. American Psychologist. 2003; 58:449–456.

    Article  PubMed  Google Scholar 

  34. Wandersman A, Florin P. Community interventions and effective prevention. American Psychologist 2003; 58:441–448.

    Article  PubMed  Google Scholar 

  35. Chinman M, Imm P, Wandersman A.Getting to outcomes 2004: promoting accountability through methods and tools for planning, implementation, and evaluation. Santa Monica, CA: RAND Corporation; 2002.

    Google Scholar 

  36. Wandersman AP, Imm P, Chinman M, et al. (2000). Getting to outcomes: a results-based approach to accountability, Evaluation and Program Planning. 2003; 23:389–395

    Article  Google Scholar 

  37. Collins J. Good to great: why some companies make the leap…and others don’t. New York: HarperCollins Publishers, Inc.; 2001.

    Google Scholar 

  38. Greenberg M. Research-based programs and lessons of implementation. In: C. Bruner, M. Greenberg, C. Guy, M. Little, H. Weiss, and L. Schorr eds. Funding what works: Exploring the role of research on effective programs and practices in government decision-making. Des Moines, IA: National Center for Service Integration Clearinghouse and the Center for Schools and Communities; 2001; 7–18.

    Google Scholar 

  39. Weiss H. (2001). Making progress: learning and public accountability. In: C Bruner, M Greenberg, C Guy, M Little, L Schorr, H Weiss eds., Funding what works: exploring the role of research on effective programs and practices in government decisionmaking. [Monograph]. Des Moines, IA: National Center for Service Integration Clearinghouse and the Center for Schools and Communities, 2001; 23–30

    Google Scholar 

  40. Jobe JB, Mingay DJ. Cognitive research improves questionnaires. American Journal of Public Health. 1989; 79:1053–1055.

    Article  PubMed  CAS  Google Scholar 

  41. Nunnally JC, Bernstein IH. Psychometric theory 3 rd ed, New York: McGraw-Hill, 1994.

    Google Scholar 

  42. Greenbaum PE, Wang W, Boothroyd R, et al. Multilevel confirmatory factor analysis of the systems of care implementation survey (SOCIS). Journal of Behavioral Health Services and Research. 2011; 38(3).

    Article  Google Scholar 

  43. Kraemer HC, Measelle JR, Ablow JC, et al. A new approach to integrating data from multiple informants in psychiatric assessment and research: mixing and matching contexts and perspectives. American Journal of Psychiatry. 2003; 160:1566–1577.

    Article  PubMed  Google Scholar 

  44. Henry GT, McTaggart MJ, McMillan JH. Establishing benchmarks for outcome indicators: a statistical approach to developing performance standards. Evaluation Review. 1992; 16:131–150.

    Article  Google Scholar 

  45. ICF Macro (2011). Children’s Mental Health Initiative (CMHI): National evaluation of the comprehensive community mental health services for children and their families program. Retrieved from: http://www.macrointernational.com/projects/cmhi/default.aspx

  46. Kutash K, Greenbaum P, Wang W, et al. Levels of system of care implementation: a national benchmarking study. Journal of Behavioral Health Services and Research. 2011; 38(3).

    Article  Google Scholar 

  47. Center for Mental Health Services (CMHS). (2002). Cooperative agreements for the Comprehensive Community Mental Health Services for Children and Their Families Program, Guidance for Applicants (GFA) (No. SM-02-002 part I—programmatic guidance). Rockville, MD: Substance Abuse and Mental Health Services Administration. Also available online from http://alt.samhsa.gov/grants/content/2002/2002grants.htm

  48. Hernandez M, Gomez A. (2002). System of Care Practice Review. Tampa: University of South Florida, Department of Child and Family Studies. Available at: http://cfs.fmhi.usf.edu/tread/misc/socpr.htm. Accessed on 4 Jan 2005.

  49. ORC Macro. System-of-care assessment 2001: Site visitor training manual. Atlanta, GA: ORC Macro; 2001.

  50. Walker JS, Koroloff N, Schutte K. Implementing high-quality collaborative individualized service/support planning: Necessary conditions. Portland, OR: Research and Training Center on Family Support and Children’s Mental Health, Portland State University, 2003

    Google Scholar 

  51. Ringel JS, Sturm R. National estimates of mental health utilization and expenditures for children in 1998, Journal of Behavioral Health Services & Research. 2001; 28:319–333.

    Article  CAS  Google Scholar 

  52. US Department of Health and Human Services. Mental health: A report of the Surgeon General. Rockville, MD: US Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health; 1999

    Google Scholar 

  53. Copeland MA. Leadership of inquiry: building and sustaining capacity for school improvement. Educational Evaluation & Policy Analysis. 2003; 25:375–395.

    Article  Google Scholar 

  54. Duchnowski AJ, Kutash K, Oliveira B. A systematic examination of school improvement activities that include special education. Remedial and Special Education. 2004; 25:117–129.

    Article  Google Scholar 

  55. Friedman RM. (1996). New forms of leadership. In: The case for kids. Tampa: University of South Florida, Department of Child and Family Studies, 19–20

    Google Scholar 

  56. Hodges S, Hernandez M, Nesman T, et al. Creating change and keeping it real: How excellent child-serving organizations carry out their goals. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, Research and Training Center for Children's Mental Health, 2002

    Google Scholar 

  57. Alter C, Murty S. Logic modeling: a tool for teaching practice evaluation. Journal of Social Work Education. 1997; 33:103–117.

    Google Scholar 

  58. Aronson SR, Mutchler SE, Pan DT. Theories of change: making programs accountable and making sense of program accountability. Austin, TX: Southwest Educational Development Laboratory, 1998.

    Google Scholar 

  59. Chen HT. Theory-driven evaluations. Newbury Park, CA: Sage, 1990.

    Google Scholar 

  60. Connell JP, Kubisch AC. Applying a theory of change approach to the evaluation of comprehensive community initiatives: progress, prospects, and problems. In: K Fulbright-Anderson, A Kubisch, JP Connell eds., New approaches to evaluating community initiatives: theory, measurement, and analysis. Washington, DC: The Aspen Institute, 1998; 15–44.

    Google Scholar 

  61. Hernandez M, Hodges S. Crafting logic models for systems of care: ideas into action. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, Department of Child and Family Studies, 2003

    Google Scholar 

  62. Fixsen DL, Blase KA. Taking evidence-based programs to scale. Paper presented at the National Association of State Mental Health Program Directors Conference, Arlington, VA, 2004.

    Google Scholar 

  63. Fixsen DL, Naoom SF, Blase KB, et al. A review and synthesis of the literature related to implementation of programs and practices. Tampa, FL: National Implementation Research Network, Louis de la Parte Florida Mental Health Institute, University of South Florida, 2005

    Google Scholar 

  64. Rosenblatt A, Wyman N, Kingdon D, et al. Managing what you measure: creating outcome-driven systems of care for youth with serious emotional disturbances. The Journal of Behavioral Health Services and Research. 1998; 25:177–193.

    Article  CAS  Google Scholar 

  65. Fixsen DL, Blase KA. Fundamentals of program implementation. Tampa, FL: Louis de la Parte Florida Mental Health Institute, University of South Florida, 2003

    Google Scholar 

  66. Rossi P, Freeman H. Evaluation: a systematic approach, 5th edition. Newbury Park, CA: Sage Publications, Inc., 1993.

    Google Scholar 

  67. Burns BJ, Costello EJ, Angold A, et al. Children’s mental health service use across service sectors. Health Affairs. 1995; 14:147–159.

    Article  PubMed  CAS  Google Scholar 

  68. Hoagwood K, Johnson J. School psychology: a public health framework: I. From evidence-based practices to evidence-based policies. Journal of School Psychology. 2003; 41:3–21.

    Article  Google Scholar 

  69. Kutash K, Duchnowski A. Create comprehensive and collaborative systems. Journal of Emotional and Behavioral Disorders. 1997; 5:66–75.

    Article  Google Scholar 

  70. Ringeisen H, Henderson K, Hoagwood K. Context matters: schools and the “research to practice gap” in children's mental health. School Psychology Review. 2003; 32:153–168.

    Google Scholar 

  71. Armstrong JS. Strategic planning improves manufacturing performance. Long Range Planning. 1991; 24:127–129.

    Article  Google Scholar 

  72. Furlong MJ, Woodbridge MW, Sosna T, et al. Santa Barbara County multiagency integrated system-of-care project. In: M Epstein, K Kutash AJ Duchnowski eds, Outcomes for children with emotional and behavioral disorders and their families: Program and evaluation best practices 2 nd ed, Austin, TX: Pro-Ed, 2005; 329–353.

    Google Scholar 

  73. Kamradt B, Gilbertson S, Lynn N. Wraparound Milwaukee: program description and evaluation. In: M Epstein, K Kutash, AJ Duchnowski eds, Outcomes for children with emotional and behavioral disorders and their families: program and evaluation best practices 2 nd ed, Austin, TX: Pro-Ed, 2005; 307–328.

    Google Scholar 

  74. US Department of Health and Human Services. Sustainability Tool Kit. Rockville, MD: US Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, 2003

    Google Scholar 

  75. Friedman RM. Children’s mental health: a status report and call to action. Invited testimony presented at the meeting of the President’s New Freedom Commission on Mental Health, Washington, D.C., 2002.

    Google Scholar 

  76. Friedman RM. Expanding the framework for developing, operating, and sustaining local systems of care. Paper presented at the Training Institutes on Developing Local Systems of Care for Children and Adolescents with Emotional Disturbances and Their Families: Family Involvement and Cultural Competence, Washington, D.C., 2002.

  77. Friedman RM. Creating informed choice for families: The link between individualized care, data-based and value-based systems of care, and evidence-based practice. Paper presented at the 11th Annual Building on Family Strengths Conference, Portland, OR, 2004.

  78. Hoza B. Psychosocial treatment issues in the MTA: a reply to Greene and Ablon. Journal of Clinical Child Psychology. 2001; 30:126–130.

    Article  PubMed  CAS  Google Scholar 

  79. Kern L, Mantegna ME, Vorndran CM, et al. Choice of task sequence to reduce problem behaviors. Journal of Positive Behavioral Interventions. 2001; 3:3–10.

    Article  Google Scholar 

  80. Deming WE. My view of quality control in Japan, Reports of Statistical Application Research. 1975; 22:73–80.

    Google Scholar 

  81. Donabedian A. Explorations in quality assessment and monitoring: the definition of quality and approaches to its assessment. Ann Arbor, MI: Health Administration Press, 1980.

    Google Scholar 

  82. Hermann RC, Regner JL, Erickson P, et al. Developing a quality management system for behavioral health care: The Cambridge Health Alliance experience. Cambridge, MA: President and Fellow of Harvard College, 2000.

    Google Scholar 

  83. Behar L, Friedman R, Lynn N. Expanding the framework for systems of care: a study of systems with extensive provider networks. Tampa, FL: University of South Florida, The Louis de la Parte Florida Mental Health Institute, 2003

    Google Scholar 

  84. Collins D. Pretesting survey instruments: an overview of cognitive methods. Quality of Life Research. 2003; 12:229–238.

    Article  PubMed  Google Scholar 

  85. Lunn LM, Heflinger CA, Wang W, et al. Community characteristics and implementation factors associated with effective systems of care. Journal of Behavioral Health Services and Research. 2011; 38(3)

    Article  Google Scholar 

Download references

Acknowledgments

This research was partially supported by Grant H133B90004 from the Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, the National Institute for Disability and Rehabilitation Research, and the National Institute of Mental Health (NIMH) (R01 MH70680-01A). The opinions contained in this manuscript are those of the authors and do not necessarily reflect those of either the US Department of Education, the Center for Mental Health Services, SAMHSA, or NIMH.

Author information

Authors and Affiliations

  1. Department of Mental Health Law and Policy, University of South Florida, 13301 Bruce B. Down Blvd, Tampa, FL, 33612-3708, USA

    Roger A. Boothroyd PhD

  2. Department of Epidemiology and Biostatistics, University of South Florida, 13301 Bruce B. Down Blvd, Tampa, FL, 33612-3708, USA

    Paul E. Greenbaum PhD, Wei Wang PhD, Krista Kutash PhD & Robert M. Friedman PhD

Authors
  1. Roger A. Boothroyd PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paul E. Greenbaum PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wei Wang PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Krista Kutash PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Robert M. Friedman PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roger A. Boothroyd PhD.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Boothroyd, R.A., Greenbaum, P.E., Wang, W. et al. Development of a Measure to Assess the Implementation of Children’s Systems of Care: The Systems of Care Implementation Survey (SOCIS). J Behav Health Serv Res 38, 288–302 (2011). https://doi.org/10.1007/s11414-011-9239-x

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11414-011-9239-x

Keywords

Search

Navigation