Governments and health insurers often make funding decisions based on health gains from randomised controlled trials. These decisions are inherently uncertain because health gains in trials may not translate to practice owing to differences in the population, treatment use and setting. Post-market analysis of real-world data can provide additional evidence but estimates from standard matching methods may be biased when unobserved characteristics explain whether a patient is treated and their outcomes. We propose a new untreated matching approach that can reduce this bias. Our approach utilises the outcomes of contemporaneous untreated patients to improve the matching of treated and historical control patients. We assess the performance of this new approach compared to standard matching using a simulation study and demonstrate the steps required using a funding decision for prostate cancer treatments in Australia. Our simulation study shows that our new matching approach eliminates nearly all bias when unobserved treatment selection is related to outcomes, and outperforms standard matching in most scenarios. In our empirical example, standard matching overestimated survival by 15% (95% confidence interval 2–34) compared to our untreated matching approach. The health gains estimated using our approach were slightly lower than expected based on the trial evidence, but we also found evidence that in practice prescribers ceased prior therapies earlier, treated a more vulnerable population and continued treatment for longer. Our untreated matching approach offers researchers a new tool for reducing uncertainty in healthcare funding decisions using real-world data.
This is a preview of subscription content, access via your institution.
Buy single article
Instant access to the full article PDF.
Tax calculation will be finalised during checkout.
Usual care could include no treatment or a range of other active therapies prescribed at different stages in the clinical management pathway before new treatment became available.
Here, we would require the random-like allocation of patients to doctors, e.g. the quick adoption of the new treatment being the only difference in treatment practice. The method we develop in the current paper can be used for the case where the funding decision (instrument) is clearly identified and there is a switch or difference from ‘no use’ of a particular treatment pathway to ‘some use’ of a treatment pathway either across time or space.
Specifically, t0 in clinical trials is the day of randomisation when exposure to the intervention is decided, which may be the same as, or just prior to initiation of treatment.
Immortal-time or survivor bias is an extreme example where matched treated-controls die before they even would have had the opportunity to be treated .
Outcomes for censored patients must be simulated before matching on outcomes.
In our approach, some uncertainty comes from whether there are too many or too few true untreated-controls in the matched pre-funding cohort. Too many may inflate the estimated treatment effect, whereas too few may underestimate the effect, given untreated-controls may be expected to have worse outcomes in general than treated-controls.
The instrumental variables’ independence assumption is E(H|X,D) = E(H|X), where H is the health outcome, X is whether (X = 1) or not (X = 0) the individual was treated, and D is whether (D = 1) or not (D = 0) the treatment was available (i.e. the instrument). This assumption implies that E(H|X = 0, D = 0) = E(H|X = 0, D = 1) or that the instrument does not affect health outcomes for those that were unaffected by the funding decision.
For completeness, we also identified SM treated-controls from the untreated patients in the post-funding period, which results in a relative bias of 76% in the base case. This illustrates that with treatment selection using contemporaneous controls can produce very biased results.
The true average treatment effect for the treated was estimated as the average survival for the treated patient (with treatment) compared to their counterfactual survival (without treatment), using a sample size of 10,000,000.
In this scenario, UM did not fail on any of the iterations because the proportion of untreated patients in each disease category (Z) was relatively low and the CEM bins were relatively coarse (i.e. there were many potential controls).
One trial used an active comparator but is equivalent to placebo for survival given the active comparator was known not to improve survival .
Compared to clinical practice in clinical trials there can be a delay between becoming eligible and starting treatment because of the trial recruitment protocols.
We estimated the treatment effect for SM using historical controls, which is consistent with the approach taken in Sect. 4. In addition, there were not many potential contemporaneous controls in this example given a large proportion (87.8%) received further treatment after the funding decision.
The registries exclude experimental treatment, but this was unlikely to significantly bias results. Australian patients included in this analysis were unlikely to have been included in prostate cancer trials during the sample period given the trial enrolment dates and eligibility criteria [42,43,44].
We predicted survival for censored patients (we observed time on docetaxel for all patients).
Most untreated patients died shortly after ceasing docetaxel, whereas treated patients initiated one of the new treatments.
The trials excluded sicker patients with multiple comorbidities and a short life expectancy, and had strict stopping criteria [33, 39]. As the new treatments were not available prior to the trials, many trial patients had already started to receive further ‘palliative’ chemotherapy (that is, chemotherapy without evidence of improved survival).
Note that while the 95% CIs of the estimated treatment effects from SM and UM largely overlap, they are highly dependent with each other such that the 95% CIs for the difference between the SM and UM estimates are significant. We can see this in Fig. A.9 of the ESM where for each bootstrapped sample the SM estimate is nearly always larger than the UM estimate.
Based on the trial data and a crude assumption of additive survival gains for second-line and third-line treatment, the expected survival gain is ≥6 months. At censoring, 37% of patients had received two treatments but we expect a higher proportion when extrapolated corresponding to a slightly longer expected survival (i.e. 4.3 months + 0.37*4.3 months = 5.9 months). It should be noted that the impact of the funding decision may not only include the impacts of the new treatments themselves, but how their use may change the treatment pathway for those individuals (such as less time on prior treatment).
Stafinski T, McCabe CJ, Menon D. Funding the unfundable. Pharmacoeconomics. 2010;28(2):113–42.
Nordon C, Karcher H, Groenwold RH, Ankarfeldt MZ, Pichler F, Chevrou-Severac H, et al. The “efficacy-effectiveness gap”: historical background and current conceptualization. Value Health. 2016;19(1):75–81.
Ghijben P, Gu Y, Lancsar E, Zavarsek S. Revealed and stated preferences of decision makers for priority setting in health technology assessment: a systematic review. Pharmacoeconomics. 2018;36(3):323–40.
Parkinson B, Sermet C, Clement F, Crausaz S, Godman B, Garner S, et al. Disinvestment and value-based purchasing strategies for pharmaceuticals: an international review. Pharmacoeconomics. 2015;33(9):905–24.
Tuffaha HW, Scuffham PA. The Australian managed entry scheme: are we getting it right? Pharmacoeconomics. 2018;36(5):555–65.
Makady A, van Veelen A, Jonsson P, Moseley O, D’Andon A, de Boer A, et al. Using real-world data in health technology assessment (HTA) practice: a comparative study of five HTA agencies. Pharmacoeconomics. 2018;36(3):359–68.
Berger ML, Sox H, Willke RJ, Brixner DL, Eichler HG, Goettsch W, et al. Good practices for real-world data studies of treatment and/or comparative effectiveness: recommendations from the joint ISPOR-ISPE Special Task Force on Real-World Evidence in Health Care Decision Making. Pharmacoepidemiol Drug Saf. 2017;26(9):1033–9.
Visvanathan K, Levit LA, Raghavan D, Hudis CA, Wong S, Dueck A, et al. Untapped potential of observational research to inform clinical decision making: American Society of Clinical Oncology research statement. J Clin Oncol. 2017;35(16):1845–54.
Velentgas P, Dreyer NA, Nourjah P, Smith SR, Torchia MM. Developing a protocol for observational comparative effectiveness research: a user's guide: AHRQ Publication No. 12(13)-EHC099. Rochville (MD): Agency for Health Research and Quality; Jan 2013.
Clarke GM, Conti S, Wolters AT, Steventon A. Evaluating the impact of healthcare interventions using routine data. BMJ. 2019;365:l2239.
de Pouvourville G, Blin P, Karam P. The contribution of real-world evidence to cost-effectiveness analysis: case study of dabigatran etexilate in France. Eur J Health Econ. 2020;21(2):235–49.
Schilling C, Petrie D, Dowsey MM, Choong PF, Clarke P. The impact of regression to the mean on economic evaluation in quasi-experimental pre–post studies: the example of total knee replacement using data from the Osteoarthritis Initiative. Health Econ. 2017;26(12):e35-51.
Johar M. The impact of the Indonesian health card program: a matching estimator approach. J Health Econ. 2009;28(1):35–53.
The European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP). Guide on methodological standards in pharmacoepidemiology (Revision 6). EMA/95098/2010. London: European Medicines Agency; 2018.
Rothwell PM. Factors that can affect the external validity of randomised controlled trials. PLoS Clin Trials. 2006;1(1):e9.
Kennedy-Martin T, Curtis S, Faries D, Robinson S, Johnston J. A literature review on the representativeness of randomized controlled trial samples and implications for the external validity of trial results. Trials. 2015;16(1):495.
Booth C, Tannock I. Randomised controlled trials and population-based observational research: partners in the evolution of medical evidence. Br J Cancer. 2014;110(3):551.
Zarin DA, Young JL, West JC. Challenges to evidence-based medicine. Soc Psychiatry Psychiatr Epidemiol. 2005;40(1):27–35.
Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials. 2009;10(1):37.
Parkinson B, Viney R, Haas M, Goodall S, Srasuebkul P, Pearson S-A. Real-world evidence: a comparison of The Australian Herceptin Program and clinical trials of trastuzumab for HER2-positive metastatic breast cancer. Pharmacoeconomics. 2016;34(10):1039–50.
World Medical Association. World Medical Association Declaration of Helsinki: ethical principles for medical research involving human subjects. JAMA. 2013;310(20):2191–4.
Tuffaha HW, Gordon LG, Scuffham PA. Value of information analysis in healthcare: a review of principles and applications. J Med Econ. 2014;17(6):377–83.
Fleurence RL, Naci H, Jansen JP. The critical role of observational evidence in comparative effectiveness research. Health Aff. 2010;29(10):1826–33.
Rubin DB. Causal inference using potential outcomes: design, modeling, decisions. J Am Stat Assoc. 2005;100(469):322–31.
Rochon PA, Gurwitz JH, Sykora K, Mamdani M, Streiner DL, Garfinkel S, et al. Reader’s guide to critical appraisal of cohort studies: 1. Role and design. BMJ. 2005;330(7496):895–7.
Suissa S. Immortal time bias in observational studies of drug effects. Pharmacoepidemiol Drug Saf. 2007;16(3):241–9.
Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41–55.
Iacus SM, King G, Porro G. Causal inference without balance checking: coarsened exact matching. Polit Anal. 2012;20(1):1–24.
Sheetz KH, Norton EC, Regenbogen SE, Dimick JB. An instrumental variable analysis comparing medicare expenditures for laparoscopic vs open colectomy. JAMA Surg. 2017;152(10):921–9.
Cain LE, Cole SR, Greenland S, Brown TT, Chmiel JS, Kingsley L, et al. Effect of highly active antiretroviral therapy on incident AIDS using calendar period as an instrumental variable. Am J Epidemiol. 2009;169(9):1124–32.
Rascati KL, Johnsrud MT, Crismon ML, Lage MJ, Barber BL. Olanzapine versus risperidone in the treatment of schizophrenia. Pharmacoeconomics. 2003;21(10):683–97.
Tchetgen EJT, Walter S, Vansteelandt S, Martinussen T, Glymour M. Instrumental variable estimation in a survival context. Epidemiology. 2015;26(3):402–10.
De Bono JS, Oudard S, Ozguroglu M, Hansen S, Machiels J-P, Kocak I, et al. Prednisone plus cabazitaxel or mitoxantrone for metastatic castration-resistant prostate cancer progressing after docetaxel treatment: a randomised open-label trial. Lancet. 2010;376(9747):1147–54.
Australian Government. Public summary document. Cabazitaxel, July 2011. Pharmaceutical Benefits Advisory Committee Meeting, 6–8 July 2011, Sydney; 2011.
Australian Government. Public summary document. Abiraterone, November 2011. Pharmaceutical Benefits Advisory Committee Meeting, 2–4 November 2011, Melbourne; 2011.
Australian Government. Public summary document. Enzalutamide, July 2014. Pharmaceutical Benefits Advisory Committee Meeting, 9–11 July 2014, Canberra; 2014.
Australian Government. Public summary document. Abiraterone, November 2012. Pharmaceutical Benefits Advisory Committee Meeting, 7–9 November 2012, Canberra; 2012.
Bahl A, Oudard S, Tombal B, Özgüroĝlu M, Hansen S, Kocak I, et al. Impact of cabazitaxel on 2-year survival and palliation of tumour-related pain in men with metastatic castration-resistant prostate cancer treated in the TROPIC trial. Ann Oncol. 2013;24(9):2402–8.
De Bono JS, Logothetis CJ, Molina A, Fizazi K, North S, Chu L, et al. Abiraterone and increased survival in metastatic prostate cancer. N Engl J Med. 2011;364(21):1995–2005.
Fizazi K, Scher HI, Molina A, Logothetis CJ, Chi KN, Jones RJ, et al. Abiraterone acetate for treatment of metastatic castration-resistant prostate cancer: final overall survival analysis of the COU-AA-301 randomised, double-blind, placebo-controlled phase 3 study. Lancet Oncol. 2012;13(10):983–92.
Scher HI, Fizazi K, Saad F, Taplin M-E, Sternberg CN, Miller K, et al. Increased survival with enzalutamide in prostate cancer after chemotherapy. N Engl J Med. 2012;367(13):1187–97.
Beer TM, Armstrong AJ, Rathkopf DE, Loriot Y, Sternberg CN, Higano CS, et al. Enzalutamide in metastatic prostate cancer before chemotherapy. N Engl J Med. 2014;371(5):424–33.
Ryan CJ, Smith MR, De Bono JS, Molina A, Logothetis CJ, De Souza P, et al. Abiraterone in metastatic prostate cancer without previous chemotherapy. N Engl J Med. 2013;368(2):138–48.
Sternberg CN, Castellano D, Daugaard G, Géczi L, Hotte SJ, Mainwaring PN, et al. Abiraterone acetate for patients with metastatic castration-resistant prostate cancer progressing after chemotherapy: final analysis of a multicentre, open-label, early-access protocol trial. Lancet Oncol. 2014;15(11):1263–8.
Sarfati D, Gurney J, Stanley J, Lim BT, McSherry C. Development of a pharmacy-based comorbidity index for patients with cancer. Med Care. 2014;52(7):586–93.
Moreira DM, Howard LE, Sourbeer KN, Amarasekara HS, Chow LC, Cockrell DC, et al. Predicting time from metastasis to overall survival in castration-resistant prostate cancer: results from SEARCH. Clin Genitourin Cancer. 2017;15(1):60-6.e2.
Armstrong AJ, Garrett-Mayer ES, Yang Y-CO, de Wit R, Tannock IF, Eisenberger M. A contemporary prognostic nomogram for men with hormone-refractory metastatic prostate cancer: a TAX327 study analysis. Clin Cancer Res. 2007;13(21):6396–403.
Heidenreich A, Bastian PJ, Bellmunt J, Bolla M, Joniau S, van der Kwast T, et al. EAU guidelines on prostate cancer. Part II: treatment of advanced, relapsing, and castration-resistant prostate cancer. Eur Urol. 2014;65(2):467–79.
Horwich A, Parker C, de Reijke T, Kataja V, ESMO Guidelines Working Group. Prostate cancer: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann Oncol. 2013;24(Suppl. 6):vi106–14.
Horwich A, Parker C, Bangma C, Kataja V, ESMO Guidelines Working Group. Prostate cancer: ESMO clinical practice guidelines for diagnosis, treatment and follow-up. Ann Oncol. 2010;21(Suppl. 5):v129–33.
Australian Government. Public summary document. Enzalutamide, November 2015. Pharmaceutical Benefits Advisory Committee Meeting, 4–6 November 2015, Canberra; 2015.
Sandblom G, Carlsson P, Sennfält K, Varenhorst E. A population-based study of pain and quality of life during the year before death in men with prostate cancer. Br J Cancer. 2004;90(6):1163.
Wolff J, Donatz V, Klier J, Erhardt W, Dass R, Geiges G. PCN120 quality of life among German patients with metastatic castration-resistant prostate cancer. Value Health. 2012;15(7):A431.
Diels J, Hamberg P, Ford D, Price PW, Spencer M, Dass R. Mapping FACT-P to EQ-5D in a large cross-sectional study of metastatic castration-resistant prostate cancer patients. Qual Life Res. 2015;24(3):591–8.
Färkkilä N, Torvinen S, Roine RP, Sintonen H, Hänninen J, Taari K, et al. Health-related quality of life among breast, prostate, and colorectal cancer patients with end-stage disease. Qual Life Res. 2014;23(4):1387–94.
Lloyd AJ, Kerr C, Penton J, Knerer G. Health-related quality of life and health utilities in metastatic castrate-resistant prostate cancer: a survey capturing experiences from a diverse sample of UK patients. Value Health. 2015;18(8):1152–7.
National Institute for Health Clinical Excellence. Technology appraisal guidance 259: Abiraterone for castration-resistant metastatic prostate cancer previously treated with a docetaxel-containing regimen. London: National Institute for Health and Clinical Excellence; 2012.
Efron B, Tibshirani RJ. An Introduction to the Bootstrap. New York: Chapman & Hall, 1994.
The authors are grateful to the Department of Human Services, Australian Government, for providing access to the data used in this paper. The findings and views of this paper are those of the authors and are not attributed to the Department of Human Services. This paper benefited from discussion with seminar participants at Monash University, University of Queensland, University of Adelaide, University of Western Australia, and the Australian Health Economics Society Conference 2019. We would also like to thank Duncan Mortimer, Jon Karnon, Johannes Kunz and two anonymous referees who provided helpful comments on previous drafts of this paper.
The Centre for Health Economics, Monash University funded this paper.
Conflicts of interest/Competing interests
Peter Ghijben, Dennis Petrie, Silva Zavarsek, Gang Chen, and Emily Lancsar have no conflicts of interest that are directly relevant to the content of this article.
Monash University Human Research Ethics Committee granted ethics approval for this study (CF15/4240-2015001816).
Consent to participate
Consent for publication
Availability of Data and Material
This paper uses administrative patient-level data owned by the Australian Government. The data are not publicly available because they contain confidential health records on Australian patients. Access to the data requires a formal application to ‘Services Australia’.
The Stata code for the Monte Carlo simulations is provided in the ESM.
All authors contributed to the study conception and design. Material preparation and data collection were performed by PG and SZ. The methodology was developed by PG and DP, and analysis was performed by PG. The first draft of the manuscript was written by PG and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.
Below is the link to the electronic supplementary material.
About this article
Cite this article
Ghijben, P., Petrie, D., Zavarsek, S. et al. Healthcare Funding Decisions and Real-World Benefits: Reducing Bias by Matching Untreated Patients. PharmacoEconomics 39, 741–756 (2021). https://doi.org/10.1007/s40273-021-01020-x