Skip to main content
SpringerLink
Log in

Linking Data to Decision-Making: Applying Qualitative Data Analysis Methods and Software to Identify Mechanisms for Using Outcomes Data

  • Regular Article
  • Published:
The Journal of Behavioral Health Services & Research Aims and scope Submit manuscript

Abstract

A multiple case study was conducted to examine how staff in child out-of-home care programs used data from an Outcomes Management System (OMS) and other sources to inform decision-making. Data collection consisted of thirty-seven semi-structured interviews with clinicians, managers, and directors from two treatment foster care programs and two residential treatment centers, and individuals involved with developing the OMS; and observations of clinical and quality management meetings. Case study and grounded theory methodology guided analyses. The application of qualitative data analysis software is described. Results show that although staff rarely used data from the OMS, they did rely on other sources of systematically collected information to inform clinical, quality management, and program decisions. Analyses of how staff used these data suggest that improving the utility of OMS will involve encouraging staff to participate in data-based decision-making, and designing and implementing OMS in a manner that reflects how decision-making processes operate.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1

Similar content being viewed by others

References

  1. Joint Commission. Facts about ORYX: The Next Evolution in Accreditation. 2005. Available at: http://www.jcaho.org/accredited+organizations/behavioral+health+care/oryx/the++next+evolution.html, accessed December 1, 2005.

  2. U.S. Public Health Service. Report of the Surgeon General’s Conference on Children’s Mental Health: A National Action Agenda. Washington, DC: Department of Health and Human Services; 2000.

    Google Scholar 

  3. Gonzalez O, Hall J, Pandiani JA, et al. The 16 state indicator pilot grant project:selected performance indicators and implications for policy and decisionmaking. In: Manderscheid RW, Henderson MJ, eds. Mental Health, United States, 2000. Rockville, MD: Center for Mental Health Services; 2001.

    Google Scholar 

  4. Manderscheid RW, Henderson MJ, Brown DY. Status of national accountability efforts at the millennium. In: Manderscheid RW, Henderson MJ, eds. Mental Health, United States, 2000. Rockville, MD: Center for Mental Health Services; 2001.

    Google Scholar 

  5. Brodey BB, Cuffel B, McCulloh J, et al. The acceptability and effectiveness of patient-reported assessments and feedback in a managed care behavioral setting. The American Journal of Managed Care. 2005;11:774–780.

    PubMed  Google Scholar 

  6. Lambert MJ, Harmon C, Slade K, et al. Providing feedback to psychotherapists on their patient’s progress: clinical results and practical suggestions. Journal of Consulting and Clinical Psychology. 2005;61:165–174.

    Google Scholar 

  7. Lambert MJ, Hansen NB, Finch AE. Patient-focused research: using patient outcomes to enhance treatment effects. Journal of Consulting and Clinical Psychology. 2001;69:159–172.

    Article  PubMed  CAS  Google Scholar 

  8. Barwick M, Boydell KM, Cunningham CE, et al. Overview of Ontario’s screening and outcome measurement initiative in children’s mental health. The Canadian Child and Adolescent Psychiatry Review. 2004;13:105–110.

    PubMed  Google Scholar 

  9. Boydell KM, Barwick M, Ferguson HB, et al. A feasibility study to assess service providers’ perspectives regarding the use of the child and adolescent functional assessment scale in Ontario. Journal of Behavioral Health Services Research. 2005;32:105.

    Article  PubMed  Google Scholar 

  10. Beck SA, Meadowcroft P, Mason M, et al. Multi-agency outcome evaluation of children’s services: a case study. Journal of Behavioral Health Services Research. 1998;25:163–176.

    Article  PubMed  CAS  Google Scholar 

  11. Rouse LW, Toprac MG, MacCabe NA. The development of a statewide continuous evaluation system for the Texas Children’s Mental Health Plan: a total quality management approach. Journal of Behavioral Health Services Research. 1998;25:194–207.

    Article  PubMed  CAS  Google Scholar 

  12. Blank MB, Koch JR, Burkett BJ. Less was more: Virginia’s performance outcomes measurement system. Psychiatric Services. 2004;55:643–645.

    Article  PubMed  Google Scholar 

  13. Koch JR, Lewis A, McCall D. A multi-stakeholder driven model for developing an outcomes management system. Journal of Behavioral Health Services Research. 1998;25:151–162.

    Article  PubMed  CAS  Google Scholar 

  14. Savas S, Fleming WM, Bolig EE. Program specification: a precursor to program monitoring and quality improvement. A case study from Boysville of Michigan. Journal of Behavioral Health Services Research. 1998;25:208–216.

    Article  PubMed  CAS  Google Scholar 

  15. Hodges K, Wotring J. The role of monitoring outcomes in initiating implementation of evidence-based treatments at the state level. Psychiatric Services. 2004;55:396–400.

    Article  PubMed  Google Scholar 

  16. Garland AF, Kruse M, Aarons GA. Clinicians and outcomes measurement: what’s the use? Journal of Behavioral Health Services Research. 2003;30:393–405.

    Article  PubMed  Google Scholar 

  17. Hodges S, Woodbridge M, Huang LN. Creating useful information in data-rich environments. In: Hernandez M, Hodges S, eds. Developing Outcome Strategies in Children’s Mental Health. Systems of Care for Children’s Mental Health. Baltimore, MD: Paul H. Brookes Publishing Co.; 2001.

    Google Scholar 

  18. Bickman L, Rosof-Williams J, Salzer MS, et al. What information do clinicians value for monitoring adolescent client progress and outcomes? Professional Psychology: Research and Practice. 2000;31:70–74.

    Article  Google Scholar 

  19. Moorman C. An innovation adoption approach to the dissemination of health care information to consumers. In: Sechrest L, Becker TE, Rogers EM, Campbell TF, Grady ML, eds. Effective Dissemination of Clinical and Health Information. Washington, DC: Agency for Health Care Policy Research; 1994:49–68.

    Google Scholar 

  20. Frambach RT, Schillewaert N. Organization innovation adoption: a multi-level framework and determinants and opportunities for future research. Journal of Business Research. 2002;55:163–176.

    Article  Google Scholar 

  21. Rogers EM. Diffusion of Innovations. New York: The Free Press; 1995.

    Google Scholar 

  22. Glisson C. Judicial and service decisions for children entering state custody: the limited role of mental health. Social Services Review. 1996;10:257–281.

    Article  Google Scholar 

  23. U.S. DHHS. Mental Health: A Report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institute of Mental Health; 1999.

    Google Scholar 

  24. Lyons JS, McCulloch JR. Monitoring and managing outcomes in residential treatment: practice-based evidence in search of evidence-based practice. Journal of American Academy of Child & Adolescent Psychiatry. 2006;45:247–251.

    Article  Google Scholar 

  25. Lyons JS, Terry P, Martinovich Z, et al. Outcome trajectories for adolescents in residential treatment: a statewide evaluation. Journal of Child and Family Studies. 2001;10:333–345.

    Article  Google Scholar 

  26. Geertz C. The Interpretation of Cultures. New York: Basic Books; 1973.

    Google Scholar 

  27. Creswell JW. Qualitative Inquiry and Research Design. Thousand Oaks, CA: Sage; 1998.

    Google Scholar 

  28. Buston K. NUD*IST in action: its use and its usefulness in a study of chronic illness in young people. Sociological Research Online[serial online]. 1997;2. Available from: http://www.socresonline.org.uk/home.html, accessed 1/7/2004.

  29. Streider F, Ayers D. SumOne for kids—Maryland: an outcome evaluation system. Paper presented at: The American Association of Children’s Residential Centers; October 18, 1997; Minneapolis, MN.

  30. Hodges K, Wong MM. Psychometric characteristics of a multidimensional measure to assess impairment: the child and adolescent functional assessment scale. Journal of Child and Family Studies. 1996;5:445–467.

    Article  Google Scholar 

  31. Hodges K. Child and Adolescent Functional Assessment Scale (CAFAS). In: Maruish ME, ed. The Use of Psychological Testing for Treatment Planning and Outcomes Assessment. 2nd ed. Mahwah, NJ: Erlbaum; 1996:631–664.

    Google Scholar 

  32. Yin R. Case Study Research: Design and Methods. 3rd ed. Thousand Oaks, CA: Sage; 2003.

    Google Scholar 

  33. McDonnell A, Jones ML, Read S. Practical considerations in case study research: the relationship between methodology and process. Journal of Advanced Nursing. 2000;32:383–390.

    Article  PubMed  CAS  Google Scholar 

  34. Patton MQ. Qualitative Evaluation and Research Methods. 3rd ed. Thousand Oaks, CA: Sage; 2002.

    Google Scholar 

  35. Miles MB, Huberman AM. Qualitative Data Analysis. Thousand Oaks, CA: Sage; 1994.

    Google Scholar 

  36. Muhr T. User’s Manual for ATLAS.ti 5.0. Berlin: ATLAS.ti Scientific Software Development GmbH; 2004.

    Google Scholar 

  37. Weitzman EA. Software and qualitative research. In: Denzin N, Lincoln Y, eds. Handbook of Qualitative Research. 2nd ed. Thousand Oaks, CA: Sage; 2000:803–820.

    Google Scholar 

  38. Strauss A, Corbin J. Basics of Qualitative Research. 2nd ed. Thousand Oaks: Sage; 1998.

    Google Scholar 

  39. Patton MQ. Enhancing the quality and credibility of qualitative analysis. Health Services Research. 1999;34:89–208.

    Google Scholar 

  40. Valenstein M, Mitchinson A, Ronis DL, et al. Quality indicators and monitoring of mental health services: what do frontline providers think? American Journal of Psychiatry. 2004;161:146–153.

    Article  PubMed  Google Scholar 

  41. Huffman LC, Martin J, Botcheva L, et al. Practitioners' attitudes toward the use of treatment progress and outcomes data in child mental health services. Evaluation & Health Professions. 2004;27:165–188.

    Article  Google Scholar 

  42. Saptya J, Riemer M, Bickman L. Feedback to clinicians: theory, research and practice. Journal of Consulting and Clinical Psychology. 2005;61:145–153.

    Google Scholar 

  43. Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measures in routine clinical practice: lack of impact or lack of theory? Social Science & Medicine. 2005;60:833–843.

    Article  Google Scholar 

  44. March J. The Pursuit of Organizational Intelligence. Malden, MA: Blackwell; 1999.

    Google Scholar 

  45. Glisson C. The organizational context of children’s mental health services. Clinical Child and Family Psychology Review. 2002;5:233–253.

    Article  PubMed  Google Scholar 

  46. Pfaffenberger B. Microcomputer Applications in Qualitative Research. Newbury Park, CA: Sage; 1988.

    Google Scholar 

  47. Kelle U. Theory building in qualitative research and computer programs for the management of textual data. Sociological Research Online [serial online]. 1997; 2. Available from: Socresonline. Accessed January 7, 2004.

  48. Bringer JD, Johnston LH, Brackenridge CH. Maximizing transparency in a doctoral thesis: the complexities of writing about the use of QSR*NVIVO within a grounded theory study. Qualitative Research. 2004;4:247–265.

    Article  Google Scholar 

  49. Hodges K, Grunwald H. The use of propensity scores to evaluate outcomes for community clinics: identification of an exceptional home-based program. Journal of Behavioral Health Services Research. 2005;32(3):294–305.

    Article  PubMed  Google Scholar 

  50. Margison FK, Barkham M, Evans C, et al. Measurement and psychotherapy: evidence-based practice and practice-based evidence. British Journal of Psychiatry. 2000;177:123–130.

    Article  PubMed  CAS  Google Scholar 

  51. Bickman L, Karver MS, Schut JA. Clinician reliability and accuracy in judging appropriate level of care. Journal of Consulting and Clinical Psychology. 1997;65:515–520.

    Article  PubMed  CAS  Google Scholar 

  52. Lambert MJ, Hawkins EJ. Measuring outcome in professional practice: considerations in selecting and using brief outcome instruments. Professional Psychology: Research and Practice. 2004;35:492–499.

    Article  Google Scholar 

  53. Harmon C, Hawkins EJ, Lambert MJ, et al. Improving outcomes for poorly responding clients: the use of clinical support tools and feedback to clients. Journal of Consulting and Clinical Psychology. 2005;61:175–185.

    Google Scholar 

  54. Berwick DM. Continuous improvement as an ideal in health care. New England Journal of Medicine. 1989;320:1450–1454.

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank the participants for sharing their thoughts on issues that affect the children they serve and the organizations they work in. We would like to acknowledge the assistance of Susan Berkowitz of Westat Corporation in helping develop the semi-structured field guides and guiding early stages of the project. We would like to thank Sharon Hodges and anonymous reviewers for their comments. This research was conducted while the first author was a doctoral student at Johns Hopkins Bloomberg School of Public Health in the Department of Health Policy and Management. This work was supported by National Institute of Mental Health # F31 MH68097, P50 MH43703, and T32 MH19545; Johns Hopkins University’s Charles D. Flagle Award and Ernest Lyman and Helen Ross Stebbins Award; and Canadian Institutes of Health Research funded Research on Addictions and Mental Health Policy & Services award.

Author information

Authors and Affiliations

  1. Clinical and Health Informatics Research Group, Faculty of Medicine, Department of Psychiatry, McGill University, Montreal, Canada

    Vaishali N. Patel PhD, MPH

  2. Department of Population, Family & Reproductive Health, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA

    Anne W. Riley PhD

  3. 1140 Pine Avenue West, Montreal, QC, H3A 1A3, Canada

    Vaishali N. Patel PhD, MPH

Authors
  1. Vaishali N. Patel PhD, MPH
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anne W. Riley PhD
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vaishali N. Patel PhD, MPH.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Patel, V.N., Riley, A.W. Linking Data to Decision-Making: Applying Qualitative Data Analysis Methods and Software to Identify Mechanisms for Using Outcomes Data. J Behav Health Serv Res 34, 459–474 (2007). https://doi.org/10.1007/s11414-007-9064-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11414-007-9064-4

Keywords

Search

Navigation