Developing Quality Indicators for Family Support Services in Community Team-Based Mental Health Care

  • S. Serene Olin
  • Krista Kutash
  • Michele Pollock
  • Barbara J. Burns
  • Anne Kuppinger
  • Nancy Craig
  • Frances Purdy
  • Kelsey Armusewicz
  • Jennifer Wisdom
  • Kimberly E. Hoagwood
Original Paper


Quality indicators for programs integrating parent-delivered family support services for children’s mental health have not been systematically developed. Increasing emphasis on accountability under the Affordable Care Act highlights the importance of quality-benchmarking efforts. Using a modified Delphi approach, quality indicators were developed for both program level and family support specialist level practices. These indicators were pilot tested with 21 community-based mental health programs. Psychometric properties of these indicators are reported; variations in program and family support specialist performance suggest the utility of these indicators as tools to guide policies and practices in organizations that integrate parent-delivered family support service components.


Quality indicators Delphi method Family support services Child mental health 



The authors would like to thank members of the Study Advisory Board and the Expert Panel Raters for their contributions. This study was supported by funding from the National Institute of Mental Health (Grant 1R01MH085969).


  1. Ainbinder, J. G., Blanchard, L. W., Singer, G. H., Sullivan, M. E., Powers, L. K., Marquis, J. G., et al. (1998). A qualitative study of parent to parent support for parents of children with special needs. Journal of Pediatric Psychology, 23(2), 99–109.PubMedCrossRefGoogle Scholar
  2. Blau, G. B., Caldwell, B., Fisher, S., Kuppinger, A., Levinson-Johnson, J., & Lieberman, R. (2010). The building bridges initiative: Residential and community-based providers, families, and youth coming together to improve outcomes. Child Welfare, 89(2), 21–38.PubMedGoogle Scholar
  3. Boulkedid, R., Abdoul, H., Loustau, M., Sibony, O., & Alberti, C. (2011). Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS ONE, 6(6), e20476.PubMedCentralPubMedCrossRefGoogle Scholar
  4. Bruns, E. J., Burchard, J. D., Suter, J. C., Leverentz-Brady, K., & Force, M. M. (2004). Assessing fidelity to a community-based treatment for youth: The wraparound fidelity index. Journal of Emotional and Behavioral Disorders, 12(2), 79–89.CrossRefGoogle Scholar
  5. Carney, P. A., Dietrich, A. J., Eliassen, M. S., Owen, M., & Badger, L. W. (1999). Recognizing and managing depression in primary care: A standardized patient study. Journal of Family Practice, 48, 965–972.PubMedGoogle Scholar
  6. Center for Health Care Strategies, Inc. (2012). Medicaid financing for family and youth peer support: A scan of state programs. Retrieved January 2013, from
  7. Chapman, D. J., Damio, G., Young, S., & Perez-Escamilla, R. (2004). Effectiveness of breastfeeding peer counseling in a low-income, predominantly Latina population: A randomized controlled trial. Archives of Pediatrics and Adolescent Medicine, 158(9), 897–902.PubMedCrossRefGoogle Scholar
  8. Chorpita, B. F., Reise, S., Weisz, J. R., Grubbs, K., Becker, K. D., & Krull, J. L. (2010). Evaluation of the brief problem checklist: Child and caregiver interviews to measure clinical progress. Journal of Consulting and Clinical Psychology, 78(4), 526.PubMedCrossRefGoogle Scholar
  9. Corrigan, P. W. (2006). Mental health stigma as social attribution: Implications for research methods and attitude change. Clinical Psychology: Science and Practice, 7(1), 48–67.Google Scholar
  10. Dixon, L., Stewart, B., Burland, J., Delahanty, J., Lucksted, A., & Hoffman, M. (2001). Pilot study of the effectiveness of the family-to-family education program. Psychiatric Services, 52(7), 965–967.PubMedCrossRefGoogle Scholar
  11. Fisher, E. B., Earp, J. A., Maman, S., & Zolotor, A. (2009a). Cross cultural and international adaptation of peer support for diabetes management. Family Practice, 27, 6–16.CrossRefGoogle Scholar
  12. Fisher, E. B., Strunk, R. C., Highstein, G. R., Kelley-Sykes, R., Tarr, K. L., Trinkaus, K., et al. (2009b). A randomized controlled evaluation of the effect of community health workers on hospitalization for asthma: the asthma coach. Archives of Pediatrics and Adolescent Medicine, 163(3), 225.PubMedCrossRefGoogle Scholar
  13. Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, the National Implementation Research Network (FMHI Publication #231).Google Scholar
  14. Ford, J. H., Green, C. A., Hoffman, K. A., Wisdom, J. P., Riley, K. J., Bergmann, L., et al. (2007). Process improvement needs in substance abuse treatment agencies: Results from walk-throughs of the admissions process. Journal of Substance Abuse Treatment, 33, 379–389.PubMedCentralPubMedCrossRefGoogle Scholar
  15. Gustafson, D. (2004). Institute for Healthcare Improvement information gathering tools: Walk through (IHI tool). Retrieved January 2013, from
  16. Hoagwood, K. E. (2005). Family-based services in children’s mental health: a research review and synthesis. Journal of Child Psychology and Psychiatry, 46(7), 690–713.PubMedCrossRefGoogle Scholar
  17. Hoagwood, K. E., Cavaleri, M. A., Olin, S., Burns, B. J., Slaton, E., Gruttadaro, D., et al. (2010). Family support in children’s mental health: a review and synthesis. Clinical child and family psychology review, 13(1), 1–45.PubMedCrossRefGoogle Scholar
  18. Horsky, J., Kaufman, D. R., Oppenheim, M. I., & Patel, V. L. (2003). A framework for analyzing the cognitive complexity of computer-assisted clinical ordering. Journal of Biomedical Informatics, 36(1–2), 4.PubMedCrossRefGoogle Scholar
  19. Institute of Medicine (IOM). (2000). Bridging disciplines in the brain, behavioral, and clinical sciences. Washington, DC: National Academy Press.Google Scholar
  20. Institute of Medicine (IOM). (2001). Crossing the quality chasm: A new health system for the 21st Century. Washington, DC: National Academy Press.Google Scholar
  21. Institute of Medicine (IOM). (2006). Improving the quality of health care for mental and substance-use conditions. Washington, DC: National Academy Press.Google Scholar
  22. Ireys, H. T., Chernoff, R., DeVet, K. A., & Kim, Y. (2001a). Maternal outcomes of a randomized controlled trial of a community-based support program for families of children with chronic illnesses. Archives of pediatrics & adolescent medicine, 155(7), 771.CrossRefGoogle Scholar
  23. Ireys, H. T., Chernoff, R., Stein, R. E., DeVet, K. A., & Silver, E. J. (2001b). Outcomes of community-based family-to-family support: Lessons learned from a decade of randomized trials. Children’s Services: Social Policy, Research, and Practice, 4(4), 203–216.CrossRefGoogle Scholar
  24. Ireys, H. T., Sills, E. M., Kolodner, K. B., & Walsh, B. B. (1996). A social support intervention for parents of children with juvenile rheumatoid arthritis: results of a randomized trial. Journal of Pediatric Psychology, 21(5), 633–641.PubMedCrossRefGoogle Scholar
  25. Jensen, P. S., & Hoagwood, K. E. (Eds.). (2008). Improving children’s mental health through parent empowerment: A guide to assisting families. New York: Oxford University Press.Google Scholar
  26. Koroloff, N. M., Elliott, D. J., Koren, P. E., & Friesen, B. J. (1996). Linking low-income families to children’s mental health services: An outcome study. Journal of Emotional and Behavioral Disorders, 4(1), 2–11.CrossRefGoogle Scholar
  27. Koroloff, N. M., Koren, P. E., Elliott, D. J., & Friesen, B. J. (1994). Connecting low-income families to mental health services: The role of the family associate. Journal of Emotional and Behavioral Disorders, 2(4), 240–246.CrossRefGoogle Scholar
  28. Kutash, K., Acri, M., Pollock, M., Armusewicz, K., Olin, S. S., & Hoagwood, K. E. (2013). Quality indicators for multidisciplinary team functioning in community-based children’s mental health services. Administration and Policy in Mental Health and Mental Health Services Research (this issue).Google Scholar
  29. Kutash, K., Duchnowski, A. J., Green, A. L., & Ferron, J. (2011). Supporting parents who have youth with emotional disturbances through a parent-to-parent support program: A proof of concept study using random assignment. Administration and Policy in Mental Health and Mental Health Services Research, 38, 412–427.PubMedCrossRefGoogle Scholar
  30. Lehmann, U., & Sanders, D. (2007). Community health workers: What do we know about them? The state of the evidence on programmes, activities, costs and impact on health outcomes of using community health workers. World Health Organization, 2, 1–42.Google Scholar
  31. Lewin, S. A., Dick, J., Pond, P., Zwarenstein, M., Aja, G., Van Wyk, B., et al. (2005). Lay health workers in primary and community health care. Cochrane Database Systemic Review, 25(1), CD004015.Google Scholar
  32. Linstone, H. A., & Turoff, M. (Eds.). (1975). The Delphi method: Techniques and applications (pp. 37–71). Reading, MA: Addison-Wesley Publishing Company.Google Scholar
  33. Min, S. Y., Whitecraft, J., Rothbard, A. B., & Salzer, M. S. (2007). Peer support for persons with co-occurring disorders and community tenure: A survival analysis. Psychiatric Rehabilitation Journal, 30(3), 207–213.PubMedCrossRefGoogle Scholar
  34. National Federation of Families for Children’s Mental Health. (2011). Job task analysis parent support providers. Orlando, FL: Webb, L.Google Scholar
  35. Obrochta, C., Anthony, B., Armstrong, M., Kalil, J., Hust, J., & Kernan, J. (2011). Issue brief: Family-to-family peer support: Models and evaluation. Atlanta, GA: ICF Macro, Outcomes Roundtable for Children and Families.Google Scholar
  36. Olin, S. S., Williams, N., Pollock, M., Armusewicz, K., Kutash, K., Glisson, C., et al. (2013, this issue). Quality indicators for family support services and their relationship to organizational social context. Administration and Policy in Mental Health and Mental Health Services Research, this issue. doi: 10.1007/s10488-013-0499-z.
  37. Osher, T. W., Penn, M., & Spencer, S. A. (2008). Partnerships with families for family-driven systems of care. In B. A. Stroul & G. M. Blau (Eds.), The system of care handbook: Transforming mental health services for children, youth, and families (pp. 249–273). Baltimore, MD: Paul H. Brookes Publishing.Google Scholar
  38. Pearson, C. R., Micek, M. A., Simoni, J. M., Hoff, P. D., Matediana, E., Martin, D. P., et al. (2007). Randomized control trial of peer-delivered, modified directly observed therapy for HAART in Mozambique. JAIDS Journal of Acquired Immune Deficiency Syndromes, 46(2), 238–244.CrossRefGoogle Scholar
  39. PEP team. (Revised 2012). Improving children’s mental health through parent empowerment: Trainer’s manual.Google Scholar
  40. Pincus, H., Page, A., Druss, B., Appelbaum, P., Gottlieb, G., & England, M. (2007). Can psychiatry cross the quality chasm? Improving the quality of health care for mental and substance use conditions. American Journal of Psychiatry, 164(5), 712–719.PubMedCrossRefGoogle Scholar
  41. Pincus, H. A., Spaeth-Rublee, B., & Watkins, K. E. (2011). The case for measuring quality in mental health and substance abuse care. Health Affairs, 30(4), 730–736.PubMedCrossRefGoogle Scholar
  42. Richard, A. L., Jongbloed, L. E., & MacFarlane, A. (2009). Integration of peer support workers into community mental health teams. International Journal of Psychosocial Rehabilitation, 14(1), 99–110.Google Scholar
  43. Robbins, V., Johnston, J., Barnett, H., Hobstetter, W., Kutash, K., Duchnowski, A. J., et al. (2008). Parent to parent: A synthesis of the emerging literature. Tampa, FL: University of South Florida, The Louis de la Parte Florida Mental Health Institute, Department of Child & Family Studies.Google Scholar
  44. Sai, C., Tonniges, T. F., Osterhus, E., & Taba, S. (2004). History of the medical home concept. Pediatrics, 113(Supplement 4), 1473–1478.Google Scholar
  45. Schoenwald, S. K., Chapman, J. E., Kelleher, K., Hoagwood, K. E., Landsverk, J., Stevens, J., et al. (2008). A survey of the infrastructure for children’s mental health services: Implications for the implementation of empirically supported treatments (ESTs). Administration and Policy in Mental Health and Mental Health Services Research, 35(1), 84–97.PubMedCrossRefGoogle Scholar
  46. Schoenwald, S. K., Ward, D. M., Henggeler, S. W., & Rowland, M. D. (2000). Multisystemic therapy versus hospitalization for crisis stabilization of youth: Placement outcomes 4 months postreferral. Mental Health Services Research, 2(1), 3–12.PubMedCrossRefGoogle Scholar
  47. Sells, D., Davidson, L., Jewell, C., Falzer, P., & Rowe, M. (2006). The treatment relationship in peer-based and regular case management for clients with severe mental illness. Psychiatric Services, 57(8), 1179–1184.PubMedCrossRefGoogle Scholar
  48. Stroul, B. A. (Ed.). (1996). Children’s mental health: Creating systems of care in a changing society. Baltimore: P.H. Brookes Publishing.Google Scholar
  49. Sullivan-Bolyai, S., Bova, C., Leung, K., Trudeau, A., Lee, M., & Gruppuso, P. (2010). Social support to empower parents (STEP) An intervention for parents of young children newly diagnosed with type 1 diabetes. The Diabetes Educator, 36(1), 88–97.PubMedCrossRefGoogle Scholar
  50. Sullivan-Bolyai, S., Knafl, K., Tamborlane, W., & Grey, M. (2004). Parents’ reflections on managing their children’s diabetes with insulin pumps. Journal of Nursing Scholarship, 36(4), 316–323.PubMedCrossRefGoogle Scholar
  51. Swider, S. M. (2002). Outcome effectiveness of community health workers: An integrative literature review. Public Health Nursing, 19(1), 11–20.PubMedCrossRefGoogle Scholar
  52. Teague, G. B., Mueser, K. T., & Rapp, C. A. (2012). Advances in fidelity measurement for mental health services research: Four measures. Psychiatric Services, 63(8), 765–771.PubMedCrossRefGoogle Scholar
  53. Wisdom, J., Lewandowski, R. E., Pollock, M., Acri, M., Shorter, P., Armusewicz, K., et al. (2013). What family support specialists do: Examining service delivery. Administration and Policy in Mental Health and Mental Health Services Research.Google Scholar
  54. Wisdom, J. P., Olin, S., Shorter, P., Burton, G., & Hoagwood, K. (2011). Family peer advocates: A pilot study of the content and process of service provision. Journal of Child and Family Studies, 20(6), 833–843.PubMedCentralPubMedCrossRefGoogle Scholar
  55. Zuvekas, A., Nolan, L., Tumaylle, C., & Griffin, L. (1999). Impact of community health workers on access, use of services, and patient knowledge and behavior. The Journal of ambulatory care management, 22(4), 33.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • S. Serene Olin
    • 1
  • Krista Kutash
    • 2
  • Michele Pollock
    • 1
  • Barbara J. Burns
    • 3
  • Anne Kuppinger
    • 4
  • Nancy Craig
    • 5
  • Frances Purdy
    • 7
  • Kelsey Armusewicz
    • 1
  • Jennifer Wisdom
    • 6
  • Kimberly E. Hoagwood
    • 1
    • 5
  1. 1.Department of Child and Adolescent Psychiatry, New York University Child Study CenterNew York University School of MedicineNew YorkUSA
  2. 2.Department of Child and Family Studies, Florida Mental Health InstituteUniversity of South FloridaTampaUSA
  3. 3.Department of Psychiatry and Behavioral SciencesDuke University School of MedicineDurhamUSA
  4. 4.Families Together in New York StateAlbanyUSA
  5. 5.New York State Office of Mental HealthNew YorkUSA
  6. 6.The George Washington UniversityWashingtonUSA
  7. 7.National Federation of Families for Children’s Mental HealthRockvilleUSA

Personalised recommendations