Skip to main content

Advertisement

SpringerLink
Log in

Provider-Associated Measurement Error in Routine Outcome Monitoring in Community Mental Health

  • Original Article
  • Published:
Administration and Policy in Mental Health and Mental Health Services Research Aims and scope Submit manuscript

Abstract

Despite the strengths of routine outcome monitoring (ROM) in community mental health settings, there are a number of barriers to effective implementation of ROM, including measurement error due to provider factors (e.g., training level) and non-target client factors (i.e., client characteristics which have no meaningful relationship to the outcome of interest). In this study, ROM data from 80 client-provider dyads were examined for sources of variance due to provider factors and non-target client factors. Results indicated that provider factors and non-target client factors accounted for between 9.6 and 54% of the variance in the ROM measures. Our findings supported past research that provider characteristics impact ROM, and added the novel finding that client gender, age, diagnosis, and cognition also impact ROM. Methods to increase accuracy and utility of ROM in community mental health are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1

Similar content being viewed by others

Notes

  1. Serious mental illness here refers to psychotic spectrum disorders that cause significant impairment in functioning, typically including schizophrenia spectrum disorders and mood disorders with psychotic features, as well as occasionally including trauma-induced disorders and obsessive–compulsive disorders depending on presentation and functional impact.

References

  • Amble, I., Gude, T., Stubdal, S., Andersen, B. J., & Wampold, B. E. (2015). The effect of implementing the Outcome Questionnaire-45.2 feedback system in Norway: A multisite randomized clinical trial in a naturalistic setting. Psychotherapy Research, 25(6), 669–677. https://doi.org/10.1080/10503307.2014.928756.

    Article  PubMed  Google Scholar 

  • APA Presidential Task Force on Evidence-Based Practice. (2006). Evidence-based practice in psychology. American Psychologist, 61(4), 271–285.

    Article  Google Scholar 

  • Baldwin, S. A., & Imel, Z. E. (2013). Therapist effects: Findings and methods. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (pp. 258–297). Hoboken, NJ: Wiley.

    Google Scholar 

  • Barker, S., Barron, N., McFarland, B. H., & Bigelow, D. A. (1994). A community ability scale for chronically mentally ill consumers: Part I. Reliability and validity. Community Mental Health Journal, 30, 363–383.

    Article  PubMed  CAS  Google Scholar 

  • Bijl, R. V., & Ravelli, A. (2000). Psychiatric morbidity, service use, and need for care in the general population: Results of The Netherlands Mental Health Survey and Incidence Study. American Journal of Public Health, 90(4), 602–607.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Bilsker, D., & Goldner, E. M. (2002). Routine outcome measurement by mental health-care providers: Is it worth doing? The Lancet, 360, 1689–1690.

    Article  Google Scholar 

  • Boswell, J. F., Constantino, M. J., Kraus, D. R., Bugatti, M., & Oswald, J. M. (2016). The expanding relevance of routinely collected outcome data for mental health care decision making. Administration and Policy in Mental Health, 43, 482–491.

    Article  PubMed  Google Scholar 

  • Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015). Implementing routine outcome monitoring in clinical practice: Benefits, challenges, and solutions. Psychotherapy Research, 25(1), 6–19. https://doi.org/10.1080/10503307.2013.817696.

    Article  PubMed  Google Scholar 

  • Burgess, P., Pirkis, J., & Coombs, T. (2015). Routine outcome measurement in Australia. International Review of Psychiatry, 27(4), 264–275. https://doi.org/10.3109/09540261.2014.977234.

    Article  PubMed  Google Scholar 

  • Carli, L. L. (1999). Gender, interpersonal power, and social influence. Journal of SocialIssues, 55(1), 81–99.

    Google Scholar 

  • Corcoran, R., Mercer, G., & Frith, C. (1995). Schizophrenia, symptomatology, and social inference: Investigating “theory of mind” in people with schizophrenia. Schizophrenia Research, 17, 5–13.

    Article  PubMed  CAS  Google Scholar 

  • Cradock, J., Young, A. S., & Sullivan, G. (2001). The accuracy of medical record documentation in schizophrenia. Journal of Behavioral Health Services & Research, 28(4), 456–465.

    Article  CAS  Google Scholar 

  • de Jong, K., Timman, R., Hakkaart-Van Roijen, L., Vermeulen, P., Kooiman, K., Passchier, J., & Busschbach, J. V. (2014). The effect of outcome monitoring feedback to clinicians and patients in short and long-term psychotherapy: A randomized controlled trial. Psychotherapy Research, 24(6), 629–639. https://doi.org/10.1080/10503307.2013.871079.

    Article  PubMed  Google Scholar 

  • Drake, R. E., Bond, G. R., & Essock, S. M. (2009). Implementing evidence based practices for people with schizophrenia. Schizophrenia Bulletin, 35(4), 704–713.

    Article  PubMed  PubMed Central  Google Scholar 

  • Fusar-Poli, P., Papanastasiou, E., Stahl, D., Rocchetti, M., Carpenter, W., Shergill, S., & McGuire, P. (2015). Treatments of negative symptoms in schizophrenia: Meta-analysis of 168 randomized placebo-controlled trials. Schizophrenia Bulletin, 41(4), 892–899. https://doi.org/10.1093/schbul/sbu170.

    Article  PubMed  Google Scholar 

  • Garb, H. N. (1989). Clinical judgement, clinical training, and professional experience. Psychological Bulletin, 105(3), 387–396.

    Article  PubMed  CAS  Google Scholar 

  • Gelkopf, M., Pagorek-Eshel, S., Trauer, T., & Roe, D. (2015). Routine outcome measurement in mental health service consumers: Who should provide support for the self-assessments? Evaluation and Program Planning, 50, 43–46. https://doi.org/10.1016/j.evalprogplan.2015.02.001.

    Article  PubMed  Google Scholar 

  • Gilbody, S. M., House, A. O., & Sheldon, T. A. (2001). Routinely administered questionnaires for depression and anxiety: Systemic review. BMJ, 322, 406–409.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  • Gilbody, S. M., House, A. O., & Sheldon, T. A. (2002a). Outcomes research in mental health: Systematic review. The British Journal of Psychiatry, 181, 8–16.

    Article  PubMed  Google Scholar 

  • Gilbody, S. M., House, A. O., & Sheldon, T. A. (2002b). Routine administration of Health Related Quality of Life (HRQoL) and needs assessment instruments to improve psychological outcome- a systematic review. Psychological Medicine, 32, 1345–1356.

    Article  PubMed  CAS  Google Scholar 

  • Gilbody, S. M., House, A. O., & Sheldon, T. A. (2002c). Psychiatrists in the UK do not use outcomes measures. British Journal of Psychiatry, 180, 101–103.

    Article  PubMed  Google Scholar 

  • Hatfield, D. R., & Ogles, B. M. (2004). The use of outcome measures by psychologists in clinical practice. Professional Psychology: Research & Practice, 35(5), 485–491.

    Article  Google Scholar 

  • Hatfield, D. R., & Ogles, B. M. (2007). Why some clinicians use outcome measures and others do not. Administration and Policy in Mental Health and Mental Health Services Research, 34, 283–291.

    Article  PubMed  Google Scholar 

  • Hill, C. E., Spiegel, S. B., Hoffman, M. A., Kivilighan, D. M., & Gelso, C. I. (2017). Therapist expertise in psychotherapy revisited. The Counseling Psychologist, 45(1), 7–53.

    Article  Google Scholar 

  • Hoenders, R. H. J., Bos, E. H., Bartels-Velthuis, A. A., Vollbehr, N. K., van der Ploeg, K., de Jonge, P., & de Jong, J. T. V. M. (2014). Pitfalls in the assessment, analysis, and interpretation of routine outcome monitoring (ROM) data: Results from an outpatientclinic for integrative mental health. Administration and Policy in Mental Health and Mental Health Services Research, 41(5), 647–659. https://doi.org/10.1007/s10488-013-0511-7.

    Article  PubMed  Google Scholar 

  • Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Hilden, J., Boutron, I., … Brorson, S. (2013). Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors. Canadian Medical Association Journal. https://doi.org/10.1503/cmaj.120744.

    Article  PubMed  Google Scholar 

  • Ionita, G., & Fitzpatrick, M. (2014). Bringing science to clinical practice: A Canadian survey of psychological practice and usage of progress monitoring measures. Canadian Psychology/Psychologie Canadienne, 55(3), 187–196. https://doi.org/10.1037/a0037355.

    Article  Google Scholar 

  • Ionita, G., Fitzpatrick, M., Tomaro, J., Chen, V. V., & Overington, L. (2016). Challenges of using progress monitoring measures: Insights from practicing clinicians. Journal of Counseling Psychology, 63(2), 173–182. https://doi.org/10.1037/cou0000122.

    Article  PubMed  Google Scholar 

  • Jastak, J. F., Jastak, S. R., & Wilkinson, G. S. (1993). The Wide-range achievement test, revised 2. Wilmington: Guidance Associates of Delaware.

    Google Scholar 

  • Jensen, R. E., Rothrock, N. E., DeWitt, E. M., Spiegel, B., Tucker, C. A., Crane, H. M., … Crane, P. K. (2015). The role of technical advances in the adoption and integration of patient-reported outcomes in clinical care. Medical Care, 53(2), 153–159. https://doi.org/10.1097/MLR.0000000000000289.

    Article  PubMed  PubMed Central  Google Scholar 

  • Jensen-Doss, A., & Hawley, K. M. (2010). Understanding barriers to evidence-based assessment: Clinician attitudes towards standardized assessment tools. Journal of Clinical Child and Adolescent Psychology, 39(6), 885–896.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kendrick, T., El-Gohary, M., Stuart, B., Gilbody, S., Churchill, R., Aiken, L., … Moore, M. (2016). Routine use of patientclient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. In The Cochrane Collaboration (Ed.), Cochrane database of systematic reviews. Chichester: Wiley https://doi.org/10.1002/14651858.CD011119.pub2.

  • Kenny, D. A. (1996). Models of non-independence in dyadic research. Journal of Social and Personal Relationships, 13(2), 279–294. https://doi.org/10.1177/0265407596132007.

    Article  Google Scholar 

  • Kotte, A., Hill, K. A., Mah, A. C., Korathu-Larson, P. A., Au, J. R., Izmirian, S., … Higa-McMillan, C. K. (2016). Facilitators and barriers of implementing a measurement feedback system in public youth mental health. Administration and Policy in Mental Health and Mental Health Services Research, 43(6), 861–878. https://doi.org/10.1007/s10488-016-0729-2.

    Article  PubMed  Google Scholar 

  • Krampen, G. (1991). Inventory on competence and control beliefs. Gottingen: Hogrefe.

    Google Scholar 

  • Lambert, M. J. (2003). Is it time for clinicians to routinely track patientt outcome? A meta-analysis. Clinical Psychology: Science and Practice, 10(3), 288–301. https://doi.org/10.1093/clipsy/bpg025.

    Article  Google Scholar 

  • Lang, K., Meyers, J. L., Korn, J. R., Lee, S., Sikirica, M., Crivera, C., Dirani, R., & Menzin, J. (2010). Medication adherence and hospitalization among patients with schizophrenia treated with antipsychotics. Psychiatric Services, 61, 1239–1247.

    Article  PubMed  Google Scholar 

  • Lyon, A. R., Dorsey, S., Pullmann, M., Silbaugh-Cowdin, J., & Berliner, L. (2015). Clinician use of standardized assessments following a common elements psychotherapy training and consultation program. Administration and Policy in Mental Health and Mental Health Services Research, 42(1), 47–60. https://doi.org/10.1007/s10488-014-0543-7.

    Article  PubMed  Google Scholar 

  • McHorney, C. A., & Tarlov, A. R. (1994). The use of health status measures for individual patient level applications: Problems and prospects. Quality of Life Research, 3(1), 43–44.

    Google Scholar 

  • Miller, S. D., Hubble, M. A., Chow, C., & Seidel, J. (2015). Beyond measures and monitoring: Realizing the potential of feedback-informed treatment. Psychotherapy, 52(4), 449–457.

    Article  PubMed  Google Scholar 

  • Reddy, L. F., Horan, W. P., & Green, M. F. (2015). Motivational deficits and negative symptoms in schizophrenia: Concepts and assessments. In Behavioral neuroscience of motivation (pp. 357–373). Cham: Springer. https://doi.org/10.1007/7854_2015_379.

  • Reitan, R. M., & Wolfson, D. (1985). The Halstead-Reitain neuropsychological test battery: Therapy and clinical interpretation. Tucson, AZ: Neuropsychological Press.

    Google Scholar 

  • Roe, D., Drake, R. E., & Slade, M. (2015a). Routine outcome monitoring: An international endeavour. International Review of Psychiatry, 27(4), 257–260. https://doi.org/10.3109/09540261.2015.1070552.

    Article  PubMed  Google Scholar 

  • Roe, D., Gelkopf, M., Gornemann, M. I., Baloush-Kleinman, V., & Shadmi, E. (2015b). Implementing routine outcome measurement in psychiatric rehabilitation services in Israel. International Review of Psychiatry, 27(4), 345–353. https://doi.org/10.3109/09540261.2015.1025722.

    Article  PubMed  Google Scholar 

  • Rosenheck, R., Fontana, A., & Stolar, M. (1999). Assessing quality of care: Administrative indicators and clinical outcomes in posttraumatic stress disorder. Med Care, 37, 180–1888.

    Article  PubMed  CAS  Google Scholar 

  • Sajatovic, M., Micula-Gondek, W., Tatuoka, C., & Bialko, C. (2011). The relationship of gender and gender identity to treatment adherence among individuals with bipolar disorder. Gender Medicine, 8(4), 261–268.

    Article  PubMed  PubMed Central  Google Scholar 

  • Serby, M. (2003). Psychiatric resident conceptualizations of mood and affect within the mental status examination. American Journal of Psychiatry, 160(8), 1527–1529. https://doi.org/10.1176/appi.ajp.160.8.1527.

    Article  PubMed  Google Scholar 

  • Spaulding, W., & Deogun, J. (2011). A pathway to personalization of integrated treatment: Informatics and decision science in psychiatric rehabilitation. Schizophrenia Bulletin, 37, 129–137.

    Article  Google Scholar 

  • Spreen, O., & Strauss, E. A. (1998). Compenium of neuropsychological tests: Administration, norms, and commentary (2nd ed.). New York: Oxford.

    Google Scholar 

  • Tait, L., Birchwood, M., & Trower, P. (2002). A new scale (SES) to measure engagement with community mental health services. Journal of Mental Health, 11(2), 191–198.

    Article  PubMed  Google Scholar 

  • Trauer, T., Callaly, T., & Herrman, H. (2009). Attitudes of mental health staff to routine outcome measurement. Journal of Mental Health, 18(4), 288–297. https://doi.org/10.1080/09638230701879177.

    Article  Google Scholar 

  • Ussher, J. M. (2013). Diagnosing difficult women and pathologising feminity: Genderbias in psychiatric nosology. Feminism & Psychology, 23(1), 63–69.

    Article  Google Scholar 

  • Wampold, B. E. (2015). Routine outcome monitoring: Coming of age—With the usual developmental challenges. Psychotherapy, 52(4), 458–462. https://doi.org/10.1037/pst0000037.

    Article  PubMed  Google Scholar 

  • Wang, P. S., Lane, M., Olfson, M., Pincus, H. A., Wells, K. B., & Kessler, R. C. (2005). Twelve-month use of mental health services in the United States: Results from the National Comorbidity Survey Replication. Archives of General Psychiatry, 62(6), 629–640. https://doi.org/10.1001/archpsyc.62.6.629.

    Article  PubMed  Google Scholar 

  • Wechsler, D. (2008). Wechsler adult intelligence: Technical and interpretive manual (4th ed.). San Antonio: Pearson.

    Google Scholar 

  • Wennberg, J. E. (1990). Outcomes research, cost containment, and the fear of health care rationing. The New England Journal of Medicine, 323(17), 1202–1204.

    Article  PubMed  CAS  Google Scholar 

  • Wolpert, M., Ford, T., Trustam, E., Law, D., Deighton, J., Flannery, H., & Fugard, R. J. B. (2012). Client-reported outcomes in child and adolescent mental health services (CAMHS): Use of idiographic and standardized measures. Journal of Mental Health, 21(2), 165–173. https://doi.org/10.3109/09638237.2012.664304.

    Article  PubMed  Google Scholar 

  • Young, A. S., Grusky, O., Sullivan, G., Webster, C. M., & Podus, D. (1998). The effect of provider characteristics on case management activities. Administration and Policy in Mental Health, 26(1), 20–32.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. VA San Diego Healthcare System, VISN 22 MIRECC & University of California-San Diego, 3350 La Jolla Village Drive, San Diego, CA, 92161, USA

    Emily B. H. Treichler

  2. University of Nebraska-Lincoln, Lincoln, NE, USA

    William D. Spaulding

Authors
  1. Emily B. H. Treichler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. William D. Spaulding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Emily B. H. Treichler.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Research Involving Human and Animal Rights and Informed Consent

This article does not contain any studies with animals performed by any of the authors. Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Treichler, E.B.H., Spaulding, W.D. Provider-Associated Measurement Error in Routine Outcome Monitoring in Community Mental Health. Adm Policy Ment Health 45, 780–789 (2018). https://doi.org/10.1007/s10488-018-0861-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-018-0861-2

Keywords

Search

Navigation