Applying User Input to the Design and Testing of an Electronic Behavioral Health Information System for Wraparound Care Coordination

  • Eric J. Bruns
  • Kelly L. Hyde
  • April Sather
  • Alyssa N. Hook
  • Aaron R. Lyon
Original Paper


Health information technology (HIT) and care coordination for individuals with complex needs are high priorities for quality improvement in health care. However, there is little empirical guidance about how best to design electronic health record systems and related technologies to facilitate implementation of care coordination models in behavioral health, or how best to apply user input to the design and testing process. In this paper, we describe an iterative development process that incorporated user/stakeholder perspectives at multiple points and resulted in an electronic behavioral health information system (EBHIS) specific to the wraparound care coordination model for youth with serious emotional and behavioral disorders. First, we review foundational HIT research on how EBHIS can enhance efficiency and outcomes of wraparound that was used to inform development. After describing the rationale for and functions of a prototype EBHIS for wraparound, we describe methods and results for a series of six small studies that informed system development across four phases of effort—predevelopment, development, initial user testing, and commercialization—and discuss how these results informed system design and refinement. Finally, we present next steps, challenges to dissemination, and guidance for others aiming to develop specialized behavioral health HIT. The research team’s experiences reinforce the opportunity presented by EBHIS to improve care coordination for populations with complex needs, while also pointing to a litany of barriers and challenges to be overcome to implement such technologies.


Wraparound Care coordination Health information technology Implementation Measurement feedback system Usability 



This publication was made possible in part by funding from the National Institute for Mental Health (R41 MH095516 and R42 MH095516). We would like to thank the Institute for Innovation and Implementation at the University of Maryland School of Social Work for informing and testing the functions of this system, and the many wraparound initiatives and behavioral health organizations that participated in user testing. Thanks also to Ricki Mudd for help in manuscript preparation.


  1. Aaron, E., & Carroll, M. D. (2015). How health information technology is failing to achieve its full potential. Pediatrics, 116(6), 1506–1512.Google Scholar
  2. Abouzahra, M., & Tan, J. (2013). The role of mobile technology in enhancing the use of personal health records. ICHITA-2013 TRANSACTIONS, 9.Google Scholar
  3. Abraham, J. M., McCullough, J. S., Parente, S. T., & Gaynor, M. S. (2011). Prevalence of electronic health records in US hospitals. Journal of Healthcare Engineering, 2(2), 121–142.Google Scholar
  4. Ambinder, E. P. (2005). Electronic health records. Journal of Oncology Practice, 1(2), 57–63.CrossRefPubMedPubMedCentralGoogle Scholar
  5. Au, M., Simon, S., Chen, A., Lipson, D., Gimm, G., & Rich, E. (2011). In M. P. Research (Ed.). Comparative effectiveness of care coordination for adults with disabilities. Princeton, NJ: Center on Health Care Effectiveness.Google Scholar
  6. Avgar, A. C., Litwin, A. S., & Pronovost, P. J. (2012). Drivers and barriers in health IT adoption: A proposed framework. Applied Clinical Informatics, 3(4), 488–500.CrossRefPubMedPubMedCentralGoogle Scholar
  7. Barth, R. P., Lee, B. R., Lindsey, M. A., Collins, K. S., Strieder, F., Chorpita, B. F., … Sparks, J. A. (2011). Evidence-based practice at a crossroads: The timely emergence of common elements and common factors. Research on Social Work Practice, 1–12.Google Scholar
  8. Bickman, L. (2008). A measurement feedback system (MFS) is necessary to improve mental health outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 47(10), 1114–1119.CrossRefPubMedPubMedCentralGoogle Scholar
  9. Bickman, L., Kelly, S. D., Breda, C., & Andrade, A. D. (2011). Effects of routine feedback to clinicians in youth mental health outcomes: A randomized cluster design. Psychiatric Services, 62, 1423–1429.CrossRefPubMedGoogle Scholar
  10. Bond, G. R., Drake, R. E., Mueser, K. T., & Latimer, E. (2012). Assertive community treatment for people with severe mental illness. Disease Management and Health Outcomes, 9(3), 141–159.CrossRefGoogle Scholar
  11. Brooke, J. (1996). SUS—A quick and dirty usability scale. In P. W. Jordan, B. Thomas, I. L. McClelland, & B. Weerdmeester (Eds.), Usability evaluation in industry (pp. 189–194). Bristol, PA: Taylor & Francis Inc.Google Scholar
  12. Browne, D. T., Puente-Duran, S., Shlonsky, A., Thabane, L., & Verticchio, D. (2014). A randomized trial of wraparound facilitation versus usual child protection services. Research on Social Work Practice, 1–14. doi: 10.1177/1049731514549630
  13. Bruns, E. J. (2015). Wraparound is worth doing well: An evidence-based statement. In E. J. Bruns & J. S. Walker (Eds.), The resource guide to wraparound. Portland, OR: National Wraparound Initiative.Google Scholar
  14. Bruns, E. J., Burchard, J., Suter, J. C., Leverentz-Brady, K. M., & Force, M. M. (2004). Assessing fidelity to a community-based treatment for youth: The wraparound fidelity index. Journal of Emotional & Behavioral Disorders, 12(2), 79–89.CrossRefGoogle Scholar
  15. Bruns, E. J., Pullmann, M. D., Sather, A., Denby Brinson, R., & Ramey, M. (2014). Effectiveness of wraparound versus case management for children and adolescents: Results of a randomized study. Administration & Policy in Mental Health and Mental Health Services Research, 1–14. doi: 10.1007/s10488-014-0571-3
  16. Bruns, E. J., Sather, A., Pullmann, M. D., & Stambaugh, L. F. (2011). National trends in implementing wraparound: Results from the state wraparound survey. Journal of Child and Family Studies, 20(6), 726–735. doi: 10.1007/s10826-011-9535-3.CrossRefGoogle Scholar
  17. Bruns, E. J., & Suter, J. C. (2010). Summary of the wraparound evidence base. In E. J. Bruns & J. S. Walker (Eds.), The resouce guide to wraparound. Portland, OR: National Wraparound Initiative.Google Scholar
  18. Bruns, E. J., Suter, J. C., Force, M. M., & Burchard, J. D. (2005). Adherence to wraparound principles and association with outcomes. Journal of Child and Family Studies, 14(4), 521–534. doi: 10.1007/s10826-005-7186-y.CrossRefGoogle Scholar
  19. Bruns, E. J., Suter, J. C., & Leverentz-Brady, K. M. (2006). Relations between program and system variables and fidelity to the wraparound process for children and families. Psychiatric Services, 57(11), 1586–1593. doi: 10.1176/ Scholar
  20. Bruns, E. J., Suter, J. C., & Leverentz-Brady, K. M. (2008). Is it wraparound yet? Setting fidelity standards for the wraparound process. Journal of Behavioral Health Services and Research, 35(3), 240–252.CrossRefPubMedGoogle Scholar
  21. Bruns, E. J., Walker, J. S., Bernstein, A., Daleiden, E., Pullmann, M. D., & Chorpita, B. F. (2013). Family voice with informed choice: Coordinating wraparound with research-based treatment for children and adolescents. Journal of Clinical Child & Adolescent Psychology, 43(2), 256–269. doi: 10.1080/15374416.2013.859081.CrossRefGoogle Scholar
  22. Bruns, E. J., Walker, J. S., Zabel, M., Estep, K., Matarese, M., Harburger, D., & Pires, S. A. (2010). Intervening in the lives of youth with complex behavioral health challenges and their families: The role of the wraparound process. American Journal of Community Psychology, 46(3–4), 314–331. doi: 10.1007/s10464-010-9346-5.CrossRefPubMedPubMedCentralGoogle Scholar
  23. Buntin, M. B., Burke, M. F., Hoaglin, M. C., & Blumenthal, D. (2011). The benefits of health information technology: A review of the recent literature shows predominantly positive results. Health Affairs, 30(3), 464–471.CrossRefPubMedGoogle Scholar
  24. Burchard, J. D., Bruns, E. J., & Burchard, S. N. (2002). The wraparound process. In B. J. Burns, K. E. Hoagwood, & M. English (Eds.), Community treatment for youth: Evidence-based treatment for severe emotional and behavioral disorders (pp. 69–90). New York: Oxford University Press.CrossRefGoogle Scholar
  25. Burns, B. (2002). Reasons for hope for children and families: A perspective and overview. In B. Burns & K. Hoagwood (Eds.), Community treatment for youth: Evidence-based treatment for severe emotional and behavioral disorders. New York: Oxford University Press.CrossRefGoogle Scholar
  26. Burns, B. J., & Goldman, K. (1999). Systems of care: Promising practices in children's mental health (Vol. IV). Washington, DC: Center for Effective Collaboration and Practice.Google Scholar
  27. Carlier, I. V., Meuldijk, D., Van Vliet, I. M., Van Fenema, E., Van der Wee, N. J., & Zitman, F. G. (2010). Routine outcome monitoring and feedback on physical or mental health status: Evidence and theory. Journal of Evaluation in Clinical Practice, 18, 104–110. doi: 10.1111/j.1365-2753.2010.01543.x.CrossRefPubMedGoogle Scholar
  28. Center for Health Care Strategies (2011, March). Fact sheet: Care management entities: A primer. Hamilton, NJ.Google Scholar
  29. Center for Medicare and Medicaid Services. (2010). Connecting kids to coverage: 2010 CHIPRA Annual Report. Rockville, MD.Google Scholar
  30. Cimino, J. J. (2013). Improving the electronic health—record: Are clinicians getting what they wished for? Journal of the American Medical Association, 309(10), 991–992.CrossRefPubMedGoogle Scholar
  31. Cohen, D. (2014). Effect of the exclusion of behavioral health from Health Information Technology (HIT) legislation on the future of integrated health care. The Journal of Behavioral Health Services & Research, 1–5.Google Scholar
  32. Cooper, J. L., Aratani, Y., Knitzer, J., Douglas-Hall, A., Masi, R., Banghart, P., & Dababnah, S. (2008). Unclaimed children revisited: The status of children’s mental health policy in the United States. New York: National Center for Children. in Poverty.Google Scholar
  33. Courage, C., & Baxter, K. (2005). Understanding your users: A practical guide to user requirements methods, tools, and techniques. San Francisco, CA: Morgan Kaufmann.Google Scholar
  34. Cox, K., Baker, D., & Wong, M. A. (2010). Wraparound retrospective: Factors predicting positive outcomes. Journal of Emotional & Behavioral Disorders, 18(1), 3–13. doi: 10.1177/1063426609336955.CrossRefGoogle Scholar
  35. de Beurs, E., den Hollander-Gijsman, M. E., van Rood, Y. R., van der Wee, N. J. A., Giltay, E. J., van Noorden, M. S., & Zitman, F. G. (2011). Routine outcome monitoring in the Netherlands: Practical experiences with a web-based strategy for the assessment of treatment outcome in clinical practice. Clinical Psychology & Psychotherapy, 18(1), 1–12. doi: 10.1002/cpp.696.CrossRefGoogle Scholar
  36. de Jong, K., van Sluis, P., Nugter, M. A., Heiser, W. J., & Spinhoven, P. (2012). Understanding the differential impact of outcome monitoring: Therapist variables that moderate feedback effects in a randomized clinical trial. Psychotherapy Research, 22(4), 464–474.CrossRefPubMedGoogle Scholar
  37. Druss, B. G., Ji, X., Glick, G., & von Esenwein, S. A. (2014). Randomized trial of an electronic personal health record for patients with serious mental illnesses. The American Journal of Psychiatry, 171(3), 360–368. doi: 10.1176/appi.ajp.2013.13070913.CrossRefPubMedGoogle Scholar
  38. Effland, V. S., Walton, B. A., & McIntyre, J. S. (2011). Connecting the dots: Stages of implementation, wraparound fidelity and youth outcomes. Journal of Child and Family Studies, 20(6), 736–746. doi: 10.1007/s10826-011-9541-5.CrossRefGoogle Scholar
  39. Fleming, N. S., Culler, S. D., McCorkle, R., Becker, E. R., & Ballard, D. J. (2011). The financial and nonfinancial costs of implementing electronic health records in primary care practices. Health Affairs, 30(3), 481–489.CrossRefPubMedGoogle Scholar
  40. Ford, E. W., Menachemi, N., & Phillips, M. T. (2006). Predicting the adoption of electronic health records by physicians: When will health care be paperless? Journal of the American Medical Informatics Association, 13(1), 106–112. doi: 10.1197/jamia.M1913.CrossRefPubMedPubMedCentralGoogle Scholar
  41. Glasgow, R. E., Kaplan, R. M., Ockene, J. K., Fisher, E. B., & Emmons, K. M. (2012). Patient-reported measures of psychosocial issues and health behavior should be added to electronic health records. Health Affairs, 31(3), 497–504. doi: 10.1377/hlthaff.2010.1295.CrossRefPubMedGoogle Scholar
  42. Grimes, K., Schulz, M., Cohen, S., Mullin, B., Lehar, S., & Tien, S. (2011). Pursuing cost-effectiveness in mental health service delivery for youth with complex needs. Journal of Mental Health Policy and Economics, 14, 73–86.PubMedGoogle Scholar
  43. Hale, A. (2008). Building databases and MIS to support wraparound implementation. In E. J. Bruns & J. S. Walker (Eds.), The resource guide to wraparound. Portland, OR: National Wraparound Initiative, Research and Training Center for Family Support and Children’s Mental Health.Google Scholar
  44. HL7 International. (2007, March 20, 2015). Meaningful use regulations. Retrieved April, 2015, from
  45. Holzner, B., Giesinger, J., Pinggera, J., Zugal, S., Schopf, F., Oberguggenberger, A., & Rumpold, G. (2012). The computer-based health evaluation software (CHES): A software for electronic patient-reported outcome monitoring. BMC Medical Informatics and Decision Making, 12(1), 126.CrossRefPubMedPubMedCentralGoogle Scholar
  46. Hoyt, R. E., & Adler, K. G. (2013). Electronic health records. In Health informatics: A practical guide (6th ed.).Google Scholar
  47. Institute of Medicine. (2003). Committee on identifying priority areas for quality improvement. Priority areas for national action: Transforming health care quality. Washington, DC: National Academies Press.Google Scholar
  48. International Organization for Standardization. (1998). ISO standards: Standards in usability and user-centered design. Standards dealing with product usage characteristics: Guidance on usability.Google Scholar
  49. Katehakis, D. G., & Tsiknakis, M. (2006). Electronic health record. Wiley Encyclopedia of Biomedical Engineering. doi: 10.1002/9780471740360.ebs1440.Google Scholar
  50. Kaufman, D., Roberts, W. D., Merrill, J., Lai, T. Y., & Bakken, S. (2006). Applying an evaluation framework for health information system design, development, and implementation. Nursing Research, 55(2), S37–S42.CrossRefPubMedGoogle Scholar
  51. Kellermann, A. L., & Jones, S. S. (2013). What it will take to achieve the as-yet-unfulfilled promises of health information technology. Health Affairs, 32(1), 63–68.CrossRefPubMedGoogle Scholar
  52. Krall, M. A., & Sittig, D. F. (2002). Clinician’s assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements (pp. 400–404).Google Scholar
  53. Lambert, M. J., Whipple, J. L., Hawkins, E. J., Vermeersch, D. A., Nielsen, S. L., & Smart, D. W. (2003). Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology: Science and Practice, 10(3), 288–301.Google Scholar
  54. Lewis, C. H. (1982). Using the “Thinking Aloud” method. In Cognitive interface design (Technical report). IBM. RC-9265.Google Scholar
  55. Lyon, A. R., Knaster Wasse, J., Ludwig, K., Zachry, M., Bruns, E. J., Unützer, J., & McCauley, E. (2015). The contextualized technology adaptation process (CTAP): Optimizing health information technology to improve mental health systems. Administration and Policy in Mental Health and Mental Health Services Research. doi: 10.1007/s10488-015-0637-x.
  56. Lyons, J. S., Griffin, E., Fazio, M., & Lyons, M. B. (1999). Child and adolescent needs and strengths: An information integration tool for children and adolescents with mental health challenges (CANS-MH), manual. Chicago: Buddin Praed Foundation, 558 Willow Rd., Winnetka, IL 60093.Google Scholar
  57. McDonald, K., Sundaram, V., Bravata, D., Lewis, R., Lin, N., Kraft, S., … Owens, D. (2007). Care coordination. Vol 7 In K. Shojania, K. McDonald, R. Wachter & D. Owens (Eds.), Closing the quality gap: A critical analysis of quality ImprovementStrategies. Technical review 9 (Prepared by the Stanford University-UCSF Evidence-basedPractice Center under contract 290-02-0017). AHRQ Publication No. 04(07)-0051-7. Rockville, MD: Agency for Healthcare Research and Quality.Google Scholar
  58. McGinn, C. A., Grenier, S., Duplantie, J., Shaw, N., Sicotte, C., Mathieu, L., … Gagnon, M. P. (2011). Comparison of user groups’ perspectives of barriers and facilitators to implementing electronic health records: a systematic review. BMC Medicine 9(1), 46.Google Scholar
  59. Michel-Verkerke, M. B., & Spil, T. A. M. (2013). The USE IT-adoption-model to predict and evaluate adoption of information and communication technology in healthcare. Methods of Information in Medicine, 52(6), 475–483.CrossRefPubMedGoogle Scholar
  60. Morrow, B. (2013, January). Electronic information exchange: Elements that matter for children in foster care: State Policy Advocacy and Reform Center.Google Scholar
  61. ONCHIT. (2011). Federal health information technology strategic plan, 2011–2015. Washington, DC.Google Scholar
  62. Perou, R., Bitsko, R. H., Blumberg, S. J., Pastor, P., Ghandour, R. M., Gfroerer, J. C., & Schieve, L. A. (2013). Mental health surveillance among children—United States, 2005–2011. Atlanta, GA: Centers for Disease Control and Prevention.Google Scholar
  63. Pires, S. A., Grimes, K. E., Allen, K. D., Gilmer, T., & Mahadevan, R. M. (2013). Faces of Medicaid: Examining children’s behavioral health service utilization and expenditures. Hamlington, NJ: Center for Health Care Strategies Inc.Google Scholar
  64. Prasad, V., & Ioannidis, J. P. (2014). Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices. Implement Science, 9(1), 5908–5909.Google Scholar
  65. Scott, K., & Lewis, C. C. (2015). Using measurement-based care to enhance any treatment. Cognitive and Behavioral Practice.Google Scholar
  66. Shimokawa, K., Lambert, M., & Smart, D. (2010). Enhancing treatment outcomes of patients at risk of treatment failure: Meta-analytic and mega-analytic review of a psychotherapy quality assurance system. Journal of Consulting and Clinical Psychology, 78(3), 298–311.CrossRefPubMedGoogle Scholar
  67. Sidorov, J. (2006). It ain’t necessarily so: The electronic health record and the unlikely prospect of reducing health care costs. Health Affairs (Millwood)., 25(4), 1079–1085.CrossRefPubMedGoogle Scholar
  68. Southam-Gerow, M. A., Daleiden, E. L., Chorpita, B. F., Bae, C., Mitchell, C. C., Faye, M., & Alba, M. (2013). MAPping Los Angeles County: Taking an evidence-informed model of mental health care to scale. Journal of Clinical Child & Adolescent Psychology, 43, 190–200. doi: 10.1080/15374416.2013.833098.CrossRefGoogle Scholar
  69. Steinhubl, S. R., Muse, E. D., & Topol, E. J. (2015). The emerging field of mobile health. Science Translational Medicine, 7(283), 283rv3–283rv3.Google Scholar
  70. Stroul, B. A., & Friedman, R. M. (1996). The system of care concept and philosophy. Children’s mental health: Creating systems of care in a changing society, 3–21.Google Scholar
  71. Surgeon General, U. S. (2001). Youth violence: A report of the Surgeon General. Washington, DC: Department of Health and Human Services.Google Scholar
  72. Suter, J. C., & Bruns, E. J. (2009). Effectiveness of the wraparound process for children with emotional and behavioral disorders: A meta-analysis. Clinical Child and Family Psychology Review, 12(4), 336–351. doi: 10.1007/s10567-009-0059-y.CrossRefPubMedGoogle Scholar
  73. Tang, P. C., Ash, J. S., Bates, D. W., Overhage, J. M., & Sands, D. Z. (2006). Personal health records: Definitions, benefits, and strategies for overcoming barriers to adoption. Journal of the American Medical Informatics Association, 13(2), 121–126. doi: 10.1197/jamia.M2025.CrossRefPubMedPubMedCentralGoogle Scholar
  74. Tolan, P. H., & Dodge, K. A. (2005). Children’s mental health as a primary care and concern: A system for comprehensive support and service. American Psychologist, 60(6), 601–614.CrossRefPubMedPubMedCentralGoogle Scholar
  75. Urdapilleta, O., Wang, Y., Varghese, R., Kim, G., Busam, S., & Palmisano, C. (2011). National evaluation of the Medicaid demonstration wavier: Home- and community-based alternatives to psychiatric residential treatment facilitites (pp. 1–166). IMPAQ International, LLC.Google Scholar
  76. U.S. Public Health Services. (2001). Youth violence: A report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services Administration, Center for Mental Health Services, National Institutes of Health, National Institutes of Mental Health.Google Scholar
  77. VanDenBerg, J. E., & Grealish, E. M. (1996). Individualized services and supports through the wraparound prcoess: Philosophy and procedures. Journal of Child and Family Studies, 5(1), 15.CrossRefGoogle Scholar
  78. Walker, J. S., & Bruns, E. J. (2006). Building on practice-based evidence: Using expert perspectives to define the wraparound process. Psychiatric Services, 57(11), 1579–1585.CrossRefPubMedGoogle Scholar
  79. Walker, J. S., Bruns, E. J., & Penn, M. (2008). Individualized services in systems of care: The wraparound process. In B. A. Stroul & G. M. Blau (Eds.), The system of care handbook: Transforming mental health services for children, youth, and families. Baltimore, MD: Paul H. Brookes Publishing Company.Google Scholar
  80. Walker, J. S., & Matarese, M. (2011). Basing organization of the wraparound workforce on a research-based theory of change. Journal of Child and Family Studies.Google Scholar
  81. Walker, J. S., & Sanders, B. (2011). The community supports for wraparound inventory: An assessment of the implementation context for wraparound. Journal of Child and Family Studies, 20(6), 747–757. doi: 10.1007/s10826-010-9432-1.
  82. Washington State Institute for Public Policy. (2012). Inventory of evidence-based, research-based, and promising practices. Olympia, WA: University of Washington, Evidence-Based Practice Institute.Google Scholar
  83. Yoe, J. T., Ryan, F. N., & Bruns, E. J. (2011). Mental health service expenditures among youth before and after enrollment into wraparound Maine: A descriptive study. Report on Emotional & Behavioral Disorders in Youth, 11(3), 61–66.Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Eric J. Bruns
    • 1
    • 3
  • Kelly L. Hyde
    • 2
  • April Sather
    • 1
  • Alyssa N. Hook
    • 1
  • Aaron R. Lyon
    • 1
  1. 1.University of Washington School of MedicineSeattleUSA
  2. 2.Social TecKnowledgy, LLCSanta FeUSA
  3. 3.Department of Psychiatry and Behavioral SciencesUniversity of WashingtonSeattleUSA

Personalised recommendations