Abstract
There is considerable debate about routine outcome monitoring (ROM) for scientific or benchmarking purposes. We discuss pitfalls associated with the assessment, analysis, and interpretation of ROM data, using data of 376 patients. 206 patients (55 %) completed one or more follow-up measurements. Mixed-model analysis showed significant improvement in symptomatology, quality of life, and autonomy, and differential improvement for different subgroups. Effect sizes were small to large, depending on the outcome measure and subgroup. Subtle variations in analytic strategies influenced effect sizes substantially. We illustrate how problems inherent to design and analysis of ROM data prevent drawing conclusions about (comparative) treatment effectiveness.
Similar content being viewed by others
Notes
In practice ROM measurements are not always completed at the intended moment; for example because the discharge of a patient is not reported timely to the ROM staff member or a respondent is slow in his or her response
References
Abdel-Khalek, A. (2006). Measuring happiness with a single-item scale. Social Behavior and Personality, 34(2), 139–150.
AMHOCN. (2013). NOCC Volume & Percentage Clinical Ratings: Australia. http://amhocn.org/analysis-reporting/nocc-volume-percentage-clinical-ratings-australia/filter/.
Appelo, M., & Harkema-Schouten, P. (2003). The Positive Outcomes List (POL) [De positieve uitkomstenlijst (PUL)]. Gedragstherapie, 36(4), 309–318.
Ashworth, M., Evans, C., & Clement, S. (2009). Measuring psychological outcomes after cognitive behaviour therapy in primary care: A comparison between a new patient-generated measure ‘‘PSYCHLOPS’’ (Psychological Outcome Profiles) and ‘‘HADS’’ (Hospital Anxiety and Depression Scale). Journal of Mental Health, 18(2), 169–177.
Bevan, G. W. Y. N., & Hood, C. H. R. I. (2006). What is measured is what matters: Targets and gaming in the english public health care system. Public Administration, 84(3), 517–538.
Bickman, L., Douglas Kelly, S., Breda, C., De Andrade, A. R., & Riemer, M. (2011). Effects of routine feedback to clinicians on mental health outcomes of youths: Results of a randomised trial. Psychiatric Services, 62(12), 1423–1429.
Bilsker, D., & Goldner, E. M. (2002). Routine outcome assessment by mental health care providers: Is it worth doing? Lancet, 360, 1689–1690.
Bosco, J. L., Silliman, R. A., Thwin, S. S., Geiger, A. M., Buist, D. S., Prout, M. N., et al. (2010). A most stubborn bias: No adjustment method fully resolves confounding by indication in observational studies. Journal of Clinical Epidemiology, 63(1), 64–74.
Brookhart, M. A., Sturmer, T., Glynn, R. J., Rassen, J., & Schneeweiss, S. (2010). Confounder control in healthcare database research; Challenges and potential approaches. Medical Care, 48(Suppl 1), 114–120.
Bruinsma, C. L., Verbraak, M. J. P. M., & Verbraak, P. (2012). ROM and benchmarking will avail transparency in mental health care [Transparantie in ggz gebaat bij ROM en benchmarking]. Tijdschrift voor Psychiatrie, 54(3), 254–256.
Burgess, P., Coombs, T., Clarke, A., Dickson, R., & Pirkis, J. (2012). Achievements in mental health outcome measurement in australia: Reflections on progress made by the Australian Mental Health Outcomes and Classification Network (AMHOCN). International Journal of Mental Health Systems, 6(1), 4.
Burgess, P., Pirkis, J., & Coombs, T. (2009). Modelling candidate effectiveness indicators for mental health services. Australian and New Zealand Journal of Psychiatry, 43(6), 531–538.
Carlier, I. V. E., Meuldijk, D., Van Vliet, I. M., Van Fenema, E., Van der Wee, N. J. A., & Zitman, F. G. (2012). Routine outcome monitoring and feedback on physical or mental health status: Evidence and theory. Journal of Evaluation in Clinical Practice, 18(1), 104–110.
Carter, T. D., Mundo, E., Parikh, S. V., & Kennedy, J. L. (2003). Early age at onset as a risk factor for poor outcome of bipolar disorder. Journal of Psychiatric Research, 37(4), 297–303.
Clark, D. M., Fairburn, C. G., & Wessely, S. (2008). Psychological treatment outcomes in routine NHS services: A commentary on Styles et al. (2007). Psychological Medicine, 38(5), 629–634.
Consortium of Academic health Centers for Integrative Medicine (CAHCIM) (2004). Retrieved 12 Oct 2012 from http://www.imconsortium.org/about/home.html.
De Beurs, E., Barendrecht, M., Flens, G., Van Dijk, E., Huijbrechts, I., & Meerdink, W. J. (2012). Progress in measuring treatment—A comparison of self reporting questionnaires [vooruitgang in de behandeling meten—Een vergelijking van vragenlijsten voor zelfrapportage]. Maandblad Geestelijke Volksgezondheid, 67(5), 259–264.
De Beurs, E., Den Hollander-Gijsman, M. E., Van Rood, Y. A., Van Der Wee, N. J. A., Giltay, E. J., Van Noorden, M. S., et al. (2011). Routine outcome monitoring in the netherlands: Practical experiences with a web-based strategy for the assessment of treatment outcome in clinical practice. Clinical Psychology Psychother, 18(1), 1–12.
Delespaul, A. E. G. (2010). The normalised t score (2). Reaction to Edwin de Beurs, ‘the normalised t score’ [De genormaliseerde T-score (2). Reactie op Edwin de Beurs, ‘De genormaliseerde T-score’]. Maandblad Geestelijke Volksgezondheid, 65(9), 841–843.
Gilbody, S. M., House, A. O., & Sheldon, T. A. (2002). Outcomes research in mental health systematic review. British Journal of Psychiatry, 181, 8–16.
Greenson, J. M., Rosenzweig, S., Halbert, S. C., Cantor, I. S., Keener, M. T., & Brainard, G. C. (2008). Integrative medicine research at an academic medical center: Patient characteristics and health-related quality-of-life outcomes. Journal of Alternative and Complementary Medicine, 14(6), 1–5.
Grilo, C. M., Stout, R. L., Markowitz, J. C., Sanislow, C. A., Ansell, E. B., Skodol, A. E., et al. (2010). Personality disorders predict relapse after remission from an episode of major depressive disorder: A 6-year prospective study. Journal of Clinical Psychiatry, 71(12), 1629–1635.
Happell, B. (2008). The value of routine outcome measurement for consumers of mental health services: Master or servant? International Journal of Social Psychiatry, 54(4), 317–327.
Hawkins, E. J., Lambert, M. J., Vermeersch, D. A., Slade, K. L., & Tuttle, K. C. (2004). The therapeutic effects of providing patient progress information to therapists and patients. Psychotherapy Research, 14(3), 308–327.
Hoenders, H. J. R., Appelo, M. T., Van Den Brink, H., Hartogs, B. M. A., & De Jong, J. T. V. M. (2011). The Dutch complementary and alternative medicine (CAM) protocol. Journal of Alternative and Complementary Medicine, 17(12), 1–5.
Höfler, M. (2005). Causal inference based on counterfactuals. BMC Medical Research Methodology, 5, 28.
Holloway, F. (2002). Outcome measurement in mental health—welcome to the revolution. British Journal of Psychiatry, 181, 1–2.
Huijbrechts, I., Appelo, M., Korrelboom, K., Van Der Heiden, C., & Bos, E. (2009). Routine outcome measurement in PsyQ: Standardisation of the 4 K’s [Routine outcome measurement binnen PsyQ: normering van de 4 K’s]. Directieve Therapie, 29(4), 243–253.
Hysong, S. J. (2009). Meta-analysis; Audit and feedback features impact effectiveness on care quality. Medical Care, 47(3), 356–363.
Iezzoni, L. I. (1996). 100 apples divided by 15 red herrings: A cautionary tale from the mid-19th century on comparing hospital mortality rates. Annals of Internal Medicine, 124(12), 1079–1085.
Lakeman, R. (2004). Standardized routine outcome measurement: Pot holes in the road to recovery. International Journal of Mental Health Nursing, 13(4), 210–215.
Lange, A., & Appelo, M. (2007). The Short Symptom List (SSL) manual [De Korte Klachten Lijst (KKL) Handleiding]. Houten: Bohn Stafleu van Loghum.
Lange, A., Schrieken, B., Van Der Ven, J. P., & Blankers, M. (2000). The Short Symptom List [De Korte Klachten Lijst]. Directieve Therapie, 20, 384–392.
Macdonald, A. (2009). Routine clinical outcomes measurement in old age psychiatry. International Psychogeriatrics, 21(6), 990–995.
MacDonald, A. J. D., & Trauer, T. (2010). Objections to routine clinical outcomes measurement in mental health services: Any evidence so far? Journal of Mental Health, 19(6), 517–522.
Massion, A. O., Dyck, I. R., Shea, M. T., Phillips, K. A., Warshaw, M. G., & Keller, M. B. (2002). Personality disorders and time to remission in generalized anxiety disorder, social phobia, and panic disorder. Archives of General Psychiatry, 59(5), 434–440.
Myklebust, M., Pradhan, E. K., & Gorenflo, D. (2008). An integrative medicine patient care model and evaluation of its outcomes: The University of Michigan experience. Journal of Alternative and Complementary Medicine, 14(7), 821–826.
Mynors-Wallis, L., & Gath, D. (1997). Predictors of treatment outcome for major depression in primary care. Psychological Medicine, 27(3), 731–736.
Nugter, M. A., & Buwalda, V. J. A. (2012). Backgrounds and uses of ROM in mental health care [Achtergronden en gebruiksmogelijkheden van ROM in de ggz]. Tijdschrift voor Psychiatrie, 54(2), 111–120.
Rush, J. A., First, M. B., & Blacker, D. (Eds.). (2000). Handbook of psychiatric measures. American Psychiatric Association: Washington DC.
Scherwitz, L. W., Cantwell, M., McHenry, P., Wood, C., & Stewart, W. (2004). A descriptive analysis of an integrative medicine clinic. Journal of Alternative and Complementary Medicine, 10(4), 651–659.
Sheehan, D. V., Harnett-Sheehan, K., & Raj, B. A. (1996). The measurement of disability. International Clinical Psychopharmacology, 11(3), 89–95.
Shrier, I., & Platt, R. W. (2008). Reducing bias through directed acyclic graphs. BMC Medical Research Methodology, 8, 70.
Slade, M., McCrone, P., Kuipers, E., Leese, M., Cahill, S., Parabiaghi, A., et al. (2006). Use of standardised outcome measures in adult mental health services. Randomised controlled trial. British Journal of Psychiatry, 189, 330–336.
Smeeding, S. J. W., Bradshaw, D. H., Kumpfer, K., Trevithick, S., & Stoddard, G. J. (2010). Outcome evaluation of the veterans affairs salt lake city integrative health clinic for chronic pain and stress related depression, anxiety and post traumatic stress disorder. Journal of Alternative and Complementary Medicine, 16(8), 823–835.
Stiles, W. B., Barkham, M., Twigg, E., Mellor-Clark, J., & Cooper, M. (2006). Effectiveness of cognitive-behavioural, person-centred and psychodynamic therapies as practised in UK national health service settings. Psychological Medicine, 36(4), 555–566.
Stiles, W. B., Leach, C., Barkham, M., Lucock, M., Iveson, S., Shapiro, D. A., et al. (2003). Early sudden gains in psychotherapy under routine clinic conditions: Practice-based evidence. Journal of Consulting and Clinical Psychology, 71(1), 14–21.
Van Der Lem, R., Van Der Wee, N. J., Van Veen, T., & Zitman, F. G. (2012). Efficacy versus effectiveness: A direct comparison of the outcome of treatment for mild to moderate depression in randomized controlled trials and daily practice. Psychotherapy and Psychosomatics, 81(4), 226–234.
Van Noorden, M. S., Van Fenema, E. M., Van der Wee, N. J. A., Zitman, F. G., & Giltay, E. J. (2012). Predicting outcome of depression using the depressive symptom profile: The Leiden routine outcome monitoring study. Depression and Anxiety, 29(6), 523–530.
Van Os, J., Kahn, R., Denys, D., Schoevers, R. A., Beekman, A. T. F., Hoogendijk, W. J. G., et al. (2012). ROM: Standard of conduct or coercive measure? [ROM: gedragsnorm of dwangmaatregel?]. Tijdschrift voor Psychiatrie, 54(3), 245–252.
Veenhoven, R. (2002). The greatest happiness for the greatest number [Het grootste geluk voor het grootste aantal]. Sociale Wetenschappen, 45(2), 1–43.
Vickers, A. J., & Altman, D. G. (2001). Analyzing controlled trials with baseline and follow up measurements. BMJ, 323, 1123–1124.
Young, A. S., Grusky, O., Jordan, D., & Belin, T. R. (2000). Routine outcome monitoring in a public mental health system: The impact of patients who leave care. Psychiatric Services, 51(1), 85–91.
Acknowledgments
The authors thank H. Berkelmans, R. Maring, P. Vellinga, M. Hoogeveen and S. Broekema for their assistance in collecting the data, and M. Appelo for his advice in the choice of design and questionnaires.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Hoenders, R.H.J., Bos, E.H., Bartels-Velthuis, A.A. et al. Pitfalls in the Assessment, Analysis, and Interpretation of Routine Outcome Monitoring (ROM) Data: Results from an Outpatient Clinic for Integrative Mental Health. Adm Policy Ment Health 41, 647–659 (2014). https://doi.org/10.1007/s10488-013-0511-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10488-013-0511-7