Skip to main content

Pitfalls in the Assessment, Analysis, and Interpretation of Routine Outcome Monitoring (ROM) Data: Results from an Outpatient Clinic for Integrative Mental Health

Administration and Policy in Mental Health and Mental Health Services Research volume 41, pages 647–659 (2014)Cite this article

Abstract

There is considerable debate about routine outcome monitoring (ROM) for scientific or benchmarking purposes. We discuss pitfalls associated with the assessment, analysis, and interpretation of ROM data, using data of 376 patients. 206 patients (55 %) completed one or more follow-up measurements. Mixed-model analysis showed significant improvement in symptomatology, quality of life, and autonomy, and differential improvement for different subgroups. Effect sizes were small to large, depending on the outcome measure and subgroup. Subtle variations in analytic strategies influenced effect sizes substantially. We illustrate how problems inherent to design and analysis of ROM data prevent drawing conclusions about (comparative) treatment effectiveness.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1

Notes

  1. In practice ROM measurements are not always completed at the intended moment; for example because the discharge of a patient is not reported timely to the ROM staff member or a respondent is slow in his or her response

References

  • Abdel-Khalek, A. (2006). Measuring happiness with a single-item scale. Social Behavior and Personality, 34(2), 139–150.

    Article  Google Scholar 

  • AMHOCN. (2013). NOCC Volume & Percentage Clinical Ratings: Australia. http://amhocn.org/analysis-reporting/nocc-volume-percentage-clinical-ratings-australia/filter/.

  • Appelo, M., & Harkema-Schouten, P. (2003). The Positive Outcomes List (POL) [De positieve uitkomstenlijst (PUL)]. Gedragstherapie, 36(4), 309–318.

    Google Scholar 

  • Ashworth, M., Evans, C., & Clement, S. (2009). Measuring psychological outcomes after cognitive behaviour therapy in primary care: A comparison between a new patient-generated measure ‘‘PSYCHLOPS’’ (Psychological Outcome Profiles) and ‘‘HADS’’ (Hospital Anxiety and Depression Scale). Journal of Mental Health, 18(2), 169–177.

    Article  Google Scholar 

  • Bevan, G. W. Y. N., & Hood, C. H. R. I. (2006). What is measured is what matters: Targets and gaming in the english public health care system. Public Administration, 84(3), 517–538.

    Article  Google Scholar 

  • Bickman, L., Douglas Kelly, S., Breda, C., De Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomised trial. Psychiatric Services, 62(12), 1423–1429.

    PubMed  Article  Google Scholar 

  • Bilsker, D., & Goldner, E. M. (2002). Routine outcome assessment by mental health care providers: Is it worth doing? Lancet, 360, 1689–1690.

    PubMed  Article  Google Scholar 

  • Bosco, J. L., Silliman, R. A., Thwin, S. S., Geiger, A. M., Buist, D. S., Prout, M. N., et al. (2010). A most stubborn bias: No adjustment method fully resolves confounding by indication in observational studies. Journal of Clinical Epidemiology, 63(1), 64–74.

    PubMed Central  PubMed  Article  Google Scholar 

  • Brookhart, M. A., Sturmer, T., Glynn, R. J., Rassen, J., & Schneeweiss, S. (2010). Confounder control in healthcare database research; Challenges and potential approaches. Medical Care, 48(Suppl 1), 114–120.

    Article  Google Scholar 

  • Bruinsma, C. L., Verbraak, M. J. P. M., & Verbraak, P. (2012). ROM and benchmarking will avail transparency in mental health care [Transparantie in ggz gebaat bij ROM en benchmarking]. Tijdschrift voor Psychiatrie, 54(3), 254–256.

    CAS  PubMed  Google Scholar 

  • Burgess, P., Coombs, T., Clarke, A., Dickson, R., & Pirkis, J. (2012). Achievements in mental health outcome measurement in australia: Reflections on progress made by the Australian Mental Health Outcomes and Classification Network (AMHOCN). International Journal of Mental Health Systems, 6(1), 4.

    PubMed Central  PubMed  Article  Google Scholar 

  • Burgess, P., Pirkis, J., & Coombs, T. (2009). Modelling candidate effectiveness indicators for mental health services. Australian and New Zealand Journal of Psychiatry, 43(6), 531–538.

    PubMed  Article  Google Scholar 

  • Carlier, I. V. E., Meuldijk, D., Van Vliet, I. M., Van Fenema, E., Van der Wee, N. J. A., & Zitman, F. G. (2012). Routine outcome monitoring and feedback on physical or mental health status: Evidence and theory. Journal of Evaluation in Clinical Practice, 18(1), 104–110.

    PubMed  Article  Google Scholar 

  • Carter, T. D., Mundo, E., Parikh, S. V., & Kennedy, J. L. (2003). Early age at onset as a risk factor for poor outcome of bipolar disorder. Journal of Psychiatric Research, 37(4), 297–303.

    PubMed  Article  Google Scholar 

  • Clark, D. M., Fairburn, C. G., & Wessely, S. (2008). Psychological treatment outcomes in routine NHS services: A commentary on Styles et al. (2007). Psychological Medicine, 38(5), 629–634.

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  • Consortium of Academic health Centers for Integrative Medicine (CAHCIM) (2004). Retrieved 12 Oct 2012 from http://www.imconsortium.org/about/home.html.

  • De Beurs, E., Barendrecht, M., Flens, G., Van Dijk, E., Huijbrechts, I., & Meerdink, W. J. (2012). Progress in measuring treatment—A comparison of self reporting questionnaires [vooruitgang in de behandeling meten—Een vergelijking van vragenlijsten voor zelfrapportage]. Maandblad Geestelijke Volksgezondheid, 67(5), 259–264.

    Google Scholar 

  • De Beurs, E., Den Hollander-Gijsman, M. E., Van Rood, Y. A., Van Der Wee, N. J. A., Giltay, E. J., Van Noorden, M. S., et al. (2011). Routine outcome monitoring in the netherlands: Practical experiences with a web-based strategy for the assessment of treatment outcome in clinical practice. Clinical Psychology Psychother, 18(1), 1–12.

    Article  Google Scholar 

  • Delespaul, A. E. G. (2010). The normalised t score (2). Reaction to Edwin de Beurs, ‘the normalised t score’ [De genormaliseerde T-score (2). Reactie op Edwin de Beurs, ‘De genormaliseerde T-score’]. Maandblad Geestelijke Volksgezondheid, 65(9), 841–843.

    Google Scholar 

  • Gilbody, S. M., House, A. O., & Sheldon, T. A. (2002). Outcomes research in mental health systematic review. British Journal of Psychiatry, 181, 8–16.

    PubMed  Article  Google Scholar 

  • Greenson, J. M., Rosenzweig, S., Halbert, S. C., Cantor, I. S., Keener, M. T., & Brainard, G. C. (2008). Integrative medicine research at an academic medical center: Patient characteristics and health-related quality-of-life outcomes. Journal of Alternative and Complementary Medicine, 14(6), 1–5.

    Google Scholar 

  • Grilo, C. M., Stout, R. L., Markowitz, J. C., Sanislow, C. A., Ansell, E. B., Skodol, A. E., et al. (2010). Personality disorders predict relapse after remission from an episode of major depressive disorder: A 6-year prospective study. Journal of Clinical Psychiatry, 71(12), 1629–1635.

    PubMed  Article  Google Scholar 

  • Happell, B. (2008). The value of routine outcome measurement for consumers of mental health services: Master or servant? International Journal of Social Psychiatry, 54(4), 317–327.

    PubMed  Article  Google Scholar 

  • Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K. L., & Tuttle, K. C. (2004). The therapeutic effects of providing patient progress information to therapists and patients. Psychotherapy Research, 14(3), 308–327.

    Article  Google Scholar 

  • Hoenders, H. J. R., Appelo, M. T., Van Den Brink, H., Hartogs, B. M. A., & De Jong, J. T. V. M. (2011). The Dutch complementary and alternative medicine (CAM) protocol. Journal of Alternative and Complementary Medicine, 17(12), 1–5.

    Article  Google Scholar 

  • Höfler, M. (2005). Causal inference based on counterfactuals. BMC Medical Research Methodology, 5, 28.

    PubMed Central  PubMed  Article  Google Scholar 

  • Holloway, F. (2002). Outcome measurement in mental health—welcome to the revolution. British Journal of Psychiatry, 181, 1–2.

    PubMed  Article  Google Scholar 

  • Huijbrechts, I., Appelo, M., Korrelboom, K., Van Der Heiden, C., & Bos, E. (2009). Routine outcome measurement in PsyQ: Standardisation of the 4 K’s [Routine outcome measurement binnen PsyQ: normering van de 4 K’s]. Directieve Therapie, 29(4), 243–253.

    Article  Google Scholar 

  • Hysong, S. J. (2009). Meta-analysis; Audit and feedback features impact effectiveness on care quality. Medical Care, 47(3), 356–363.

    PubMed  Article  Google Scholar 

  • Iezzoni, L. I. (1996). 100 apples divided by 15 red herrings: A cautionary tale from the mid-19th century on comparing hospital mortality rates. Annals of Internal Medicine, 124(12), 1079–1085.

    CAS  PubMed  Article  Google Scholar 

  • Lakeman, R. (2004). Standardized routine outcome measurement: Pot holes in the road to recovery. International Journal of Mental Health Nursing, 13(4), 210–215.

    PubMed  Article  Google Scholar 

  • Lange, A., & Appelo, M. (2007). The Short Symptom List (SSL) manual [De Korte Klachten Lijst (KKL) Handleiding]. Houten: Bohn Stafleu van Loghum.

    Google Scholar 

  • Lange, A., Schrieken, B., Van Der Ven, J. P., & Blankers, M. (2000). The Short Symptom List [De Korte Klachten Lijst]. Directieve Therapie, 20, 384–392.

    Google Scholar 

  • Macdonald, A. (2009). Routine clinical outcomes measurement in old age psychiatry. International Psychogeriatrics, 21(6), 990–995.

    PubMed  Article  Google Scholar 

  • MacDonald, A. J. D., & Trauer, T. (2010). Objections to routine clinical outcomes measurement in mental health services: Any evidence so far? Journal of Mental Health, 19(6), 517–522.

    PubMed  Article  Google Scholar 

  • Massion, A. O., Dyck, I. R., Shea, M. T., Phillips, K. A., Warshaw, M. G., & Keller, M. B. (2002). Personality disorders and time to remission in generalized anxiety disorder, social phobia, and panic disorder. Archives of General Psychiatry, 59(5), 434–440.

    PubMed  Article  Google Scholar 

  • Myklebust, M., Pradhan, E. K., & Gorenflo, D. (2008). An integrative medicine patient care model and evaluation of its outcomes: The University of Michigan experience. Journal of Alternative and Complementary Medicine, 14(7), 821–826.

    Article  Google Scholar 

  • Mynors-Wallis, L., & Gath, D. (1997). Predictors of treatment outcome for major depression in primary care. Psychological Medicine, 27(3), 731–736.

    CAS  PubMed  Article  Google Scholar 

  • Nugter, M. A., & Buwalda, V. J. A. (2012). Backgrounds and uses of ROM in mental health care [Achtergronden en gebruiksmogelijkheden van ROM in de ggz]. Tijdschrift voor Psychiatrie, 54(2), 111–120.

    CAS  PubMed  Google Scholar 

  • Rush, J. A., First, M. B., & Blacker, D. (Eds.). (2000). Handbook of psychiatric measures. American Psychiatric Association: Washington DC.

    Google Scholar 

  • Scherwitz, L. W., Cantwell, M., McHenry, P., Wood, C., & Stewart, W. (2004). A descriptive analysis of an integrative medicine clinic. Journal of Alternative and Complementary Medicine, 10(4), 651–659.

    Article  Google Scholar 

  • Sheehan, D. V., Harnett-Sheehan, K., & Raj, B. A. (1996). The measurement of disability. International Clinical Psychopharmacology, 11(3), 89–95.

    PubMed  Article  Google Scholar 

  • Shrier, I., & Platt, R. W. (2008). Reducing bias through directed acyclic graphs. BMC Medical Research Methodology, 8, 70.

    PubMed Central  PubMed  Article  Google Scholar 

  • Slade, M., McCrone, P., Kuipers, E., Leese, M., Cahill, S., Parabiaghi, A., et al. (2006). Use of standardised outcome measures in adult mental health services. Randomised controlled trial. British Journal of Psychiatry, 189, 330–336.

    PubMed  Article  Google Scholar 

  • Smeeding, S. J. W., Bradshaw, D. H., Kumpfer, K., Trevithick, S., & Stoddard, G. J. (2010). Outcome evaluation of the veterans affairs salt lake city integrative health clinic for chronic pain and stress related depression, anxiety and post traumatic stress disorder. Journal of Alternative and Complementary Medicine, 16(8), 823–835.

    Article  Google Scholar 

  • Stiles, W. B., Barkham, M., Twigg, E., Mellor-Clark, J., & Cooper, M. (2006). Effectiveness of cognitive-behavioural, person-centred and psychodynamic therapies as practised in UK national health service settings. Psychological Medicine, 36(4), 555–566.

    PubMed  Article  Google Scholar 

  • Stiles, W. B., Leach, C., Barkham, M., Lucock, M., Iveson, S., Shapiro, D. A., et al. (2003). Early sudden gains in psychotherapy under routine clinic conditions: Practice-based evidence. Journal of Consulting and Clinical Psychology, 71(1), 14–21.

    PubMed  Article  Google Scholar 

  • Van Der Lem, R., Van Der Wee, N. J., Van Veen, T., & Zitman, F. G. (2012). Efficacy versus effectiveness: A direct comparison of the outcome of treatment for mild to moderate depression in randomized controlled trials and daily practice. Psychotherapy and Psychosomatics, 81(4), 226–234.

    PubMed  Article  Google Scholar 

  • Van Noorden, M. S., Van Fenema, E. M., Van der Wee, N. J. A., Zitman, F. G., & Giltay, E. J. (2012). Predicting outcome of depression using the depressive symptom profile: The Leiden routine outcome monitoring study. Depression and Anxiety, 29(6), 523–530.

    PubMed  Article  Google Scholar 

  • Van Os, J., Kahn, R., Denys, D., Schoevers, R. A., Beekman, A. T. F., Hoogendijk, W. J. G., et al. (2012). ROM: Standard of conduct or coercive measure? [ROM: gedragsnorm of dwangmaatregel?]. Tijdschrift voor Psychiatrie, 54(3), 245–252.

    PubMed  Google Scholar 

  • Veenhoven, R. (2002). The greatest happiness for the greatest number [Het grootste geluk voor het grootste aantal]. Sociale Wetenschappen, 45(2), 1–43.

    Google Scholar 

  • Vickers, A. J., & Altman, D. G. (2001). Analyzing controlled trials with baseline and follow up measurements. BMJ, 323, 1123–1124.

    CAS  PubMed Central  PubMed  Article  Google Scholar 

  • Young, A. S., Grusky, O., Jordan, D., & Belin, T. R. (2000). Routine outcome monitoring in a public mental health system: The impact of patients who leave care. Psychiatric Services, 51(1), 85–91.

    CAS  PubMed  Google Scholar 

Download references

Acknowledgments

The authors thank H. Berkelmans, R. Maring, P. Vellinga, M. Hoogeveen and S. Broekema for their assistance in collecting the data, and M. Appelo for his advice in the choice of design and questionnaires.

Author information

Authors and Affiliations

  1. Center for Integrative Psychiatry, Lentis, Groningen, The Netherlands

    Rogier H. J. Hoenders, Agna A. Bartels-Velthuis, Nina K. Vollbehr & Karen van der Ploeg

  2. Interdisciplinary Center Psychopathology and Emotion regulation, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands

    Elisabeth H. Bos & Peter de Jonge

  3. University Center for Psychiatry, University Medical Center Groningen, University of Groningen, Groningen, The Netherlands

    Agna A. Bartels-Velthuis

  4. Amsterdam Institute for Social Science Research (AISSR), University of Amsterdam, Amsterdam, The Netherlands

    Joop T. V. M. de Jong

  5. Boston School of Medicine, Boston, MA, USA

    Joop T. V. M. de Jong

  6. Rhodes University, Grahamstown, South Africa

    Joop T. V. M. de Jong

Authors
  1. Rogier H. J. Hoenders
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elisabeth H. Bos
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Agna A. Bartels-Velthuis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Nina K. Vollbehr
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Karen van der Ploeg
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Peter de Jonge
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Joop T. V. M. de Jong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Elisabeth H. Bos.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Hoenders, R.H.J., Bos, E.H., Bartels-Velthuis, A.A. et al. Pitfalls in the Assessment, Analysis, and Interpretation of Routine Outcome Monitoring (ROM) Data: Results from an Outpatient Clinic for Integrative Mental Health. Adm Policy Ment Health 41, 647–659 (2014). https://doi.org/10.1007/s10488-013-0511-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10488-013-0511-7

Keywords

  • Routine outcome monitoring
  • Treatment outcome
  • Integrative psychiatry
  • Bias
  • Confounding