Understanding Treatment Effect Estimates When Treatment Effects Are Heterogeneous for More Than One Outcome
Patient-centred care requires evidence of treatment effects across many outcomes. Outcomes can be beneficial (e.g. increased survival or cure rates) or detrimental (e.g. adverse events, pain associated with treatment, treatment costs, time required for treatment). Treatment effects may also be heterogeneous across outcomes and across patients. Randomized controlled trials are usually insufficient to supply evidence across outcomes. Observational data analysis is an alternative, with the caveat that the treatments observed are choices. Real-world treatment choice often involves complex assessment of expected effects across the array of outcomes. Failure to account for this complexity when interpreting treatment effect estimates could lead to clinical and policy mistakes.
Our objective was to assess the properties of treatment effect estimates based on choice when treatments have heterogeneous effects on both beneficial and detrimental outcomes across patients.
Simulation methods were used to highlight the sensitivity of treatment effect estimates to the distributions of treatment effects across patients across outcomes. Scenarios with alternative correlations between benefit and detriment treatment effects across patients were used. Regression and instrumental variable estimators were applied to the simulated data for both outcomes.
True treatment effect parameters are sensitive to the relationships of treatment effectiveness across outcomes in each study population. In each simulation scenario, treatment effect estimate interpretations for each outcome are aligned with results shown previously in single outcome models, but these estimates vary across simulated populations with the correlations of treatment effects across patients across outcomes.
If estimator assumptions are valid, estimates across outcomes can be used to assess the optimality of treatment rates in a study population. However, because true treatment effect parameters are sensitive to correlations of treatment effects across outcomes, decision makers should be cautious about generalizing estimates to other populations.
JB devised the initial concept, the simulation models, and drafted the main article. CC and MS assisted with the conceptual framework, manuscript edits, and presentation of results.
Compliance with Ethical Standards
This project was funded by the Patient-Centered Outcomes Research Institute (PCORI) under project number (ME-1303-6011).
Conflict of interest
John Brooks, Cole Chapman, and Mary Schroeder have no conflicts of interest that are directly relevant to the content of this study.
Data Availability Statement
The data from the five simulation scenarios presented are available as ZIP SAS datasets in supplementary material for this paper.
- 11.Heckman JJ, The scientific model of causality. Sociol Methodol 35, 2005. 35: p. 1-97.Google Scholar
- 17.Heckman JJ, Robb R. Alternative Methods for Evaluating the Impact of Interventions, in Longitudinal Analysis of Labor Market Data. In: Heckman JJ, Singer B (eds). 1985, Cambridge University Press: New York. p. 156–245.Google Scholar
- 18.Angrist JD, Ferandez-Val I. ExtrapoLATE-ing: external validity and overidentification in the LATE framework. Advances in Economics and Econometrics, Vol Iii: Econometrics, ed. Acemoglu D, Arellano M, Dekel E. 2013. 401–433.Google Scholar
- 19.Angrist JD, Pischke J-S. Mostly harmless econometrics: an empiricist’s companion. New Jersey: Princeton University Press; 2009.Google Scholar
- 23.Brooks JM, McClellan M, Wong HS. The marginal benefits of invasive treatments for acute myocardial infarction: Does insurance coverage matter? Inquiry-the J Health Care Organ Provis Financ. 2000;37(1):75–90.Google Scholar
- 38.Crown WH, Henk HJ, Vanness DJ. Some cautions on the use of instrumental variables estimators in outcomes research: how bias in instrumental variables estimators is affected by instrument strength, instrument contamination, and sample size. Value Health. 2011;14(8):1078–84.CrossRefPubMedGoogle Scholar
- 39.Bound J, Jaeger DA, Baker RM. Problems with instrumental variables estimation when the correlation between the instruments and the endogenous explanatory variable is weak. J Am Stat Assoc. 1995;90(430):443–50.Google Scholar
- 42.Ben-Akiva M, Lerman SR, Analysis Discrete choice. Cambridge. Massachusetts: The MIT Press; 1985.Google Scholar