Influence of Modeling Choices on Value of Information Analysis: An Empirical Analysis from a Real-World Experiment
Value of information (VOI) analysis often requires modeling to characterize and propagate uncertainty. In collaboration with a cancer clinical trial group, we integrated a VOI approach to assessing trial proposals.
This paper aims to explore the impact of modeling choices on VOI results and to share lessons learned from the experience.
After selecting two proposals (A: phase III, breast cancer; B: phase II, pancreatic cancer) for in-depth evaluations, we categorized key modeling choices relevant to trial decision makers (characterizing uncertainty of efficacy, evidence thresholds to change clinical practice, and sample size) and modelers (cycle length, survival distribution, simulation runs, and other choices). Using a $150,000 per quality-adjusted life-year (QALY) threshold, we calculated the patient-level expected value of sample information (EVSI) for each proposal and examined whether each modeling choice led to relative change of more than 10% from the averaged base-case estimate. We separately analyzed the impact of the effective time horizon.
The base-case EVSI was $118,300 for Proposal A and $22,200 for Proposal B per patient. Characterizing uncertainty of efficacy was the most important choice in both proposals (e.g. Proposal A: $118,300 using historical data vs. $348,300 using expert survey), followed by the sample size and the choice of survival distribution. The assumed effective time horizon also had a substantial impact on the population-level EVSI.
Modeling choices can have a substantial impact on VOI. Therefore, it is important for groups working to incorporate VOI into research prioritization to adhere to best practices, be clear in their reporting and justification for modeling choices, and to work closely with the relevant decision makers, with particular attention to modeling choices.
DDK and JJC conceived and designed the study, and SDR secured funding for this study. DDK and GFG were responsible for conducting empirical analyses. DDK wrote the initial manuscript and acted as overall guarantor for the overall content of this article. All authors elaborated, discussed, and approved the final version of the manuscript submitted for publication.
Compliance with Ethical Standards
Financial support for this study was provided in part by an award from the Patient-Centered Outcomes Research Institute (PCORI) [ME-1303-5889; PI: Scott Ramsey]. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, and writing and publishing the report. All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), or its Board of Governors or Methodology Committee.
Conflict of interest
DDK, GFG, CSB, AB, DLV, SDR, and JJC have no other conflicts of interest to declare in the subject matter discussed in this manuscript.
- 7.Briggs A, Sculpher M, Claxton K. Decision modelling for health economic evaluation. Oxford: Oxford University Press; 2006.Google Scholar
- 16.SWOG. About Us. 2018. https://oldweb.swog.org/Visitors/AboutUs.asp. Accessed 8 May 2018.
- 17.Bennette CS, Veenstra DL, Basu A, Baker LH, Ramsey SD, Carlson JJ. Development and evaluation of an approach to using value of information analyses for real-time prioritization decisions within SWOG, a Large Cancer Clinical Trials Cooperative Group. Med Decis Mak. 2016;36(5):641–51.CrossRefGoogle Scholar
- 21.Expected Value of Sample Information (EVSI). York: York Health Economics Consortium; 2016. http://www.yhec.co.uk/glossary/expected-value-of-sample-information-evsi/. Accessed 2 June 2019
- 26.Djulbegovic B, Kumar A, Soares HP, Hozo I, Bepler G, Clarke M, et al. Treatment success in cancer: new cancer treatment successes identified in phase 3 randomized controlled trials conducted by the National Cancer Institute-sponsored cooperative oncology groups, 1955 to 2006. Arch Intern Med. 2008;168(6):632–42.CrossRefGoogle Scholar
- 31.R Core Team R. A Language and Environment for Statistical Computing. 2018. http://www.R-project.org/. Accessed 20 Oct 2017.