Influence of Modeling Choices on Value of Information Analysis: An Empirical Analysis from a Real-World Experiment

  • David D. KimEmail author
  • Gregory F. Guzauskas
  • Caroline S. Bennette
  • Anirban Basu
  • David L. Veenstra
  • Scott D. Ramsey
  • Josh J. Carlson
Original Research Article



Value of information (VOI) analysis often requires modeling to characterize and propagate uncertainty. In collaboration with a cancer clinical trial group, we integrated a VOI approach to assessing trial proposals.


This paper aims to explore the impact of modeling choices on VOI results and to share lessons learned from the experience.


After selecting two proposals (A: phase III, breast cancer; B: phase II, pancreatic cancer) for in-depth evaluations, we categorized key modeling choices relevant to trial decision makers (characterizing uncertainty of efficacy, evidence thresholds to change clinical practice, and sample size) and modelers (cycle length, survival distribution, simulation runs, and other choices). Using a $150,000 per quality-adjusted life-year (QALY) threshold, we calculated the patient-level expected value of sample information (EVSI) for each proposal and examined whether each modeling choice led to relative change of more than 10% from the averaged base-case estimate. We separately analyzed the impact of the effective time horizon.


The base-case EVSI was $118,300 for Proposal A and $22,200 for Proposal B per patient. Characterizing uncertainty of efficacy was the most important choice in both proposals (e.g. Proposal A: $118,300 using historical data vs. $348,300 using expert survey), followed by the sample size and the choice of survival distribution. The assumed effective time horizon also had a substantial impact on the population-level EVSI.


Modeling choices can have a substantial impact on VOI. Therefore, it is important for groups working to incorporate VOI into research prioritization to adhere to best practices, be clear in their reporting and justification for modeling choices, and to work closely with the relevant decision makers, with particular attention to modeling choices.


Author Contributions

DDK and JJC conceived and designed the study, and SDR secured funding for this study. DDK and GFG were responsible for conducting empirical analyses. DDK wrote the initial manuscript and acted as overall guarantor for the overall content of this article. All authors elaborated, discussed, and approved the final version of the manuscript submitted for publication.

Compliance with Ethical Standards


Financial support for this study was provided in part by an award from the Patient-Centered Outcomes Research Institute (PCORI) [ME-1303-5889; PI: Scott Ramsey]. The funding agreement ensured the authors’ independence in designing the study, interpreting the data, and writing and publishing the report. All statements in this report, including its findings and conclusions, are solely those of the authors and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute (PCORI), or its Board of Governors or Methodology Committee.

Conflict of interest

DDK, GFG, CSB, AB, DLV, SDR, and JJC have no other conflicts of interest to declare in the subject matter discussed in this manuscript.

Supplementary material

40273_2019_848_MOESM1_ESM.docx (201 kb)
Supplementary material 1 (DOCX 200 kb)


  1. 1.
    von Niederhausern B, Guyatt GH, Briel M, Pauli-Magnus C. Academic response to improving value and reducing waste: a comprehensive framework for INcreasing QUality In patient-oriented academic clinical REsearch (INQUIRE). PLoS Med. 2018;15(6):e1002580.CrossRefGoogle Scholar
  2. 2.
    Moher D, Glasziou P, Chalmers I, Nasser M, Bossuyt PMM, Korevaar DA, et al. Increasing value and reducing waste in biomedical research: who’s listening? Lancet. 2016;387(10027):1573–86.CrossRefGoogle Scholar
  3. 3.
    Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. J Health Econ. 1999;18(3):341–64.CrossRefGoogle Scholar
  4. 4.
    Claxton K, Neumann PJ, Araki S, Weinstein MC. Bayesian value-of-information analysis. An application to a policy model of Alzheimer’s disease. Int J Technol Assess Health Care. 2001;17(1):38–55.CrossRefGoogle Scholar
  5. 5.
    Ades AE, Lu G, Claxton K. Expected value of sample information calculations in medical decision modeling. Med Decis Mak. 2004;24(2):207–27.CrossRefGoogle Scholar
  6. 6.
    Groot Koerkamp B, Spronk S, Stijnen T, Hunink MG. Value of information analyses of economic randomized controlled trials: the treatment of intermittent claudication. Value Health. 2010;13(2):242–50.CrossRefGoogle Scholar
  7. 7.
    Briggs A, Sculpher M, Claxton K. Decision modelling for health economic evaluation. Oxford: Oxford University Press; 2006.Google Scholar
  8. 8.
    Claxton K, Sculpher M, Drummond M. A rational framework for decision making by the National Institute For Clinical Excellence (NICE). Lancet. 2002;360(9334):711–5.CrossRefGoogle Scholar
  9. 9.
    Claxton KP, Sculpher MJ. Using value of information analysis to prioritise health research: some lessons from recent UK experience. Pharmacoeconomics. 2006;24(11):1055–68.CrossRefGoogle Scholar
  10. 10.
    Kim DD, Trikalinos TA, Wong JB. Leveraging cumulative network meta-analysis and value of information analysis to understand the evolving value of medical research. Med Decis Mak. 2019;39(2):119–29.CrossRefGoogle Scholar
  11. 11.
    Jackson CH, Bojke L, Thompson SG, Claxton K, Sharples LD. A framework for addressing structural uncertainty in decision models. Med Decis Mak. 2011;31(4):662–74.CrossRefGoogle Scholar
  12. 12.
    Bojke L, Claxton K, Sculpher M, Palmer S. Characterizing structural uncertainty in decision analytic models: a review and application of methods. Value Health. 2009;12(5):739–49.CrossRefGoogle Scholar
  13. 13.
    Koffijberg H, Rothery C, Chalkidou K, Grutters J. Value of information choices that influence estimates: a systematic review of prevailing considerations. Med Decis Mak. 2018;38(7):888–900.CrossRefGoogle Scholar
  14. 14.
    Steuten L, van de Wetering G, Groothuis-Oudshoorn K, Retel V. A systematic and critical review of the evolving methods and applications of value of information in academia and practice. Pharmacoeconomics. 2013;31(1):25–48.CrossRefGoogle Scholar
  15. 15.
    Scoggins JF, Ramsey SD. A national cancer clinical trials system for the 21st century: reinvigorating the NCI Cooperative Group Program. J Natl Cancer Inst. 2010;102(17):1371.CrossRefGoogle Scholar
  16. 16.
    SWOG. About Us. 2018. Accessed 8 May 2018.
  17. 17.
    Bennette CS, Veenstra DL, Basu A, Baker LH, Ramsey SD, Carlson JJ. Development and evaluation of an approach to using value of information analyses for real-time prioritization decisions within SWOG, a Large Cancer Clinical Trials Cooperative Group. Med Decis Mak. 2016;36(5):641–51.CrossRefGoogle Scholar
  18. 18.
    Carlson JJ, Kim DD, Guzauskas GF, Bennette CS, Veenstra DL, Basu A, et al. Integrating value of research into NCI Clinical Trials Cooperative Group research review and prioritization: a pilot study. Cancer Med. 2018;7(9):4251–60.CrossRefGoogle Scholar
  19. 19.
    Meltzer DO, Hoomans T, Chung JW, Basu A. Minimal modeling approaches to value of information analysis for health research. Med Decis Mak. 2011;31(6):E1–22.CrossRefGoogle Scholar
  20. 20.
    Sanders GD, Neumann PJ, Basu A, Brock DW, Feeny D, Krahn M, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: second panel on cost-effectiveness in health and medicine. JAMA. 2016;316(10):1093–103.CrossRefGoogle Scholar
  21. 21.
    Expected Value of Sample Information (EVSI). York: York Health Economics Consortium; 2016. Accessed 2 June 2019
  22. 22.
    Ryen L, Svensson M. The willingness to pay for a quality adjusted life year: a review of the empirical literature. Health Econ. 2015;24(10):1289–301.CrossRefGoogle Scholar
  23. 23.
    Greenland S. Modeling and variable selection in epidemiologic analysis. Am J Public Health. 1989;79(3):340–9.CrossRefGoogle Scholar
  24. 24.
    Greenland S, Daniel R, Pearce N. Outcome modelling strategies in epidemiology: traditional methods and basic alternatives. Int J Epidemiol. 2016;45(2):565–75.CrossRefGoogle Scholar
  25. 25.
    Lee PH. Is a cutoff of 10% appropriate for the change-in-estimate criterion of confounder identification? J Epidemiol. 2014;24(2):161–7.CrossRefGoogle Scholar
  26. 26.
    Djulbegovic B, Kumar A, Soares HP, Hozo I, Bepler G, Clarke M, et al. Treatment success in cancer: new cancer treatment successes identified in phase 3 randomized controlled trials conducted by the National Cancer Institute-sponsored cooperative oncology groups, 1955 to 2006. Arch Intern Med. 2008;168(6):632–42.CrossRefGoogle Scholar
  27. 27.
    Djulbegovic B, Kumar A, Glasziou PP, Perera R, Reljic T, Dent L, et al. New treatments compared to established treatments in randomized trials. Cochrane Database Syst Rev. 2012;10:MR000024.PubMedPubMedCentralGoogle Scholar
  28. 28.
    Basu A, Meltzer D. Decision criterion and value of information analysis: optimal aspirin dosage for secondary prevention of cardiovascular events. Med Decis Mak. 2018;38(4):427–38.CrossRefGoogle Scholar
  29. 29.
    Johnson N, Lilford RJ, Brazier W. At what level of collective equipoise does a clinical trial become ethical? J Med Ethics. 1991;17(1):30–4.CrossRefGoogle Scholar
  30. 30.
    Latimer NR. Survival analysis for economic evaluations alongside clinical trials—extrapolation with patient-level data: inconsistencies, limitations, and a practical guide. Med Decis Mak. 2013;33(6):743–54.CrossRefGoogle Scholar
  31. 31.
    R Core Team R. A Language and Environment for Statistical Computing. 2018. Accessed 20 Oct 2017.
  32. 32.
    Kim DD, Neumann PJ. Comparative Modeling to Inform Health Policy Decisions: A Step Forward. Ann Intern Med. 2019;. Scholar
  33. 33.
    Philips Z, Claxton K, Palmer S. The half-life of truth: what are appropriate time horizons for research decisions? Med Decis Mak. 2008;28(3):287–99.CrossRefGoogle Scholar
  34. 34.
    Bojke L, Grigore B, Jankovic D, Peters J, Soares M, Stein K. Informing reimbursement decisions using cost-effectiveness modelling: a guide to the process of generating elicited priors to capture model uncertainties. Pharmacoeconomics. 2017;35(9):867–77.CrossRefGoogle Scholar
  35. 35.
    Neumann PJ, Kim DD, Trikalinos TA, Sculpher MJ, Salomon JA, Prosser LA, et al. Future directions for cost-effectiveness analyses in health and medicine. Med Decis Mak. 2018;38(7):767–77.CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Center for the Evaluation of Value and Risk in Health, Institute for Clinical Research and Health Policy StudiesTufts Medical CenterBostonUSA
  2. 2.Department of PharmacyUniversity of WashingtonSeattleUSA
  3. 3.Flatiron HealthNew YorkUSA
  4. 4.Hutchinson Institute for Cancer Outcomes ResearchFred Hutchinson Cancer Research CenterSeattleUSA

Personalised recommendations