Informing Reimbursement Decisions Using Cost-Effectiveness Modelling: A Guide to the Process of Generating Elicited Priors to Capture Model Uncertainties


In informing decisions, utilising health technology assessment (HTA), expert elicitation can provide valuable information, particularly where there is a less-developed evidence-base at the point of market access. In these circumstances, formal methods to elicit expert judgements are preferred to improve the accountability and transparency of the decision-making process, help reduce bias and the use of heuristics, and also provide a structure that allows uncertainty to be expressed. Expert elicitation is the process of transforming the subjective and implicit knowledge of experts into their quantifiable expressions. The use of expert elicitation in HTA is gaining momentum, and there is particular interest in its application to diagnostics, medical devices and complex interventions such as in public health or social care. Compared with the gathering of experimental evidence, elicitation constitutes a reasonably low-cost source of evidence. Given its inherent subject nature, the potential biases in elicited evidence cannot be ignored and, due to its infancy in HTA, there is little guidance to the analyst wishing to conduct a formal elicitation exercise. This article attempts to summarise the stages of designing and conducting an expert elicitation, drawing on key literature and examples, most of which are not in HTA. In addition, we critique their applicability to HTA, given its distinguishing features. There are a number of issues that the analyst should be mindful of, in particular the need to appropriately characterise the uncertainty associated with model inputs and the fact that there are often numerous parameters required, not all of which can be defined using the same quantities. This increases the need for the elicitation task to be as straightforward as possible for the expert to complete.

This is a preview of subscription content, log in to check access.


  1. 1.

    Hunger T, et al. Using expert opinion in health technology assessment: a guideline review. Int J Technol Assessm Health Care. 2016;32(3):131–9.

    Article  Google Scholar 

  2. 2.

    Grigore B, et al. Methods to elicit probability distributions from experts: a systematic review of reported practice in health technology assessment. Pharmacoeconomics. 2013;31(11):991–1003.

    PubMed  Article  Google Scholar 

  3. 3.

    Hadorn D, et al. Use of expert knowledge elicitation to estimate parameters in health economic decision models. Int J Technol Assess Health Care. 2014;30(4):461–8.

    PubMed  Article  Google Scholar 

  4. 4.

    Iglesias CP, et al. Reporting guidelines for the use of expert judgement in model-based economic evaluations. Pharmacoeconomics. 2016;34(11):1161–72.

    PubMed  Article  Google Scholar 

  5. 5.

    Hart A, O’Hagan A, Quigley J, Bolger F. Training course on steering an expert knowledge elicitation. Final report. EFSA Supporting publ. 2016;13(5):1009E. doi:10.2903/sp.efsa.2016.EN-1009.

    Google Scholar 

  6. 6.

    O’Hagan A, et al. Uncertain judgements: eliciting experts’ probabilities. New York: Wiley; 2006.

    Google Scholar 

  7. 7.

    Hora SC, Von Winterfeldt D. Nuclear waste and future societies: a look into the deep future. Technol Forecast Soc Change. 1997;56(2):155–70.

    Article  Google Scholar 

  8. 8.

    Jenkinson D. The elicitation of probabilities: a review of the statistical literature. Bayesian elicitation of experts’ probabilities (BEEP). Sheffield: Sheffield University; 2005.

    Google Scholar 

  9. 9.

    Soares MO, et al. Methods to elicit experts’ beliefs over uncertain quantities: application to a cost effectiveness transition model of negative pressure wound therapy for severe pressure ulceration. Stat Med. 2011;30(19):2363–80.

    PubMed  Article  Google Scholar 

  10. 10.

    Clemen RT, Winkler RL. Aggregating probability distributions. In: Edwards W, Miles Jr R, Von Winterfeldt D, editors. Advances in decision analysis: from foundations to applications. Cambridge: Cambridge University Press; 2007. p. 154–76.

    Google Scholar 

  11. 11.

    French S. Aggregating expert judgement. Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. Serie A. Matematicas. 2011;105(1):181–206.

    Google Scholar 

  12. 12.

    Johnson SR, et al. Methods to elicit beliefs for Bayesian priors: a systematic review. J Clin Epidemiol. 2010;63(4):355–69.

    PubMed  Article  Google Scholar 

  13. 13.

    Kattan MW, et al. The wisdom of crowds of doctors: their average predictions outperform their individual ones. Med Dec Mak. 2016;36(4):536–40.

    Article  Google Scholar 

  14. 14.

    Cooke RM, Goossens LHJ. Expert judgement elicitation for risk assessments of critical infrastructures. J Risk Res. 2004;7(6):643–56.

    Article  Google Scholar 

  15. 15.

    Knol AB, et al. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environ Health Glob Access Sci Source. 2010;9:19.

    Google Scholar 

  16. 16.

    Kadane J, Wolfson LJ. Experiences in elicitation. J R Stat Soc Ser D Stat. 1998;47(1):3–19.

    Article  Google Scholar 

  17. 17.

    Renooij S, Witteman C. Talking probabilities: communicating probabilistic information with words and numbers. Int J Approx Reas. 1999;22(3):169–94.

    Article  Google Scholar 

  18. 18.

    Bruine de Bruin W, et al. What number is “fifty-fifty”? Redistributing excessive 50% responses in elicited probabilities. Risk Anal. 2002;22(4):713–23.

    PubMed  Article  Google Scholar 

  19. 19.

    Claxton K, et al. Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra. Health Econ. 2005;14(4):339–47.

    PubMed  Article  Google Scholar 

  20. 20.

    Leal J, Wordsworth S, Legood R, Blair E. Eliciting expert opinion for economic models: an applied example. Value Health. 2007;10(3):195–203.

    PubMed  Article  Google Scholar 

  21. 21.

    Grigore B, et al. A comparison of two methods for expert elicitation in health technology assessments. BMC Med Res Methodol. 2016;16:85.

    PubMed  PubMed Central  Article  Google Scholar 

  22. 22.

    Pibouleau L, Chevret S. An internet-based method to elicit experts’ beliefs for Bayesian priors: a case study in intracranial stent evaluation. Int J Technol Assess Health Care. 2014;30(4):1–8.

    Article  Google Scholar 

  23. 23.

    Bojke L, et al. Eliciting distributions to populate decision analytic models. Value Health. 2010;13(5):557–64.

    PubMed  Article  Google Scholar 

  24. 24.

    McKenna C, et al. Enhanced external counterpulsation for the treatment of stable angina and heart failure: a systematic review and economic evaluation. Health Technol Assess. 2009;13(24):1–90, iii–iv, ix–xi.

    PubMed  Article  Google Scholar 

  25. 25.

    Speight PM, et al. The cost-effectiveness of screening for oral cancer in primary care. Health Technol Assess. 2006;10(14):1–144, iii–iv.

    CAS  PubMed  Article  Google Scholar 

  26. 26.

    Van Noortwijk JM, et al. Expert judgment in maintenance optimization. IEEE Trans Reliab. 1992;41(3):427–32.

    Article  Google Scholar 

  27. 27.

    Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Publ Health. 2005;27:281–91.

    Article  Google Scholar 

  28. 28.

    Knol AB, et al. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environ Health. 2010;9:19.

    PubMed  PubMed Central  Article  Google Scholar 

  29. 29.

    Baker E, et al. Facing the experts: survey mode and expert elicitation. Fondazione Eni Enrico Mattei. Nora di Lavoro; 2014.

  30. 30.

    Morris DE, Oakley JE, Crowe JA. A web-based tool for eliciting probability distributions from experts. Environ Model Softw. 2014;52:1–4.

    Article  Google Scholar 

  31. 31.

    Expert Judgement Network 2016. Accessed 13 Jun 2017.

  32. 32.

    Elfadaly FG, Garthwaite PH. Eliciting Dirichlet and Connor–Mosimann prior distributions for multinomial models. TEST. 2013;22(4):628–46.

    Article  Google Scholar 

  33. 33.

    Garthwaite PH, et al. Use of expert knowledge in evaluating costs and benefits of alternative service provisions: a case study. Int J Technol Assess Health Care. 2008;24(3):350–7.

    PubMed  Article  Google Scholar 

  34. 34.

    WikiBooks. Cognitive science: an introduction/biases and reasoning heuristics. 2016. Accessed 3 May 2017.

  35. 35.

    Tversky A, Kahneman D. The framing of decisions and the psychology of choice. Science. 1981;4481:453–8.

    Article  Google Scholar 

  36. 36.

    Garthwaite PH, Kadane JB, O’Hagan A. Statistical methods for eliciting probability distributions. J Am Stat Assoc. 2005;100(470):680–701.

    CAS  Article  Google Scholar 

  37. 37.

    Montibeller G, von Winterfeldt D. Cognative and motivational biases in decison and risk analysis. Risk Anal. 2015;35(7):1230–51.

    PubMed  Article  Google Scholar 

  38. 38.

    Kynn M. The ‘heuristics and biases’ bias in expert elicitation. J R Stat Soc Ser A Stat Soc. 2008;171(1):239–64.

    Google Scholar 

  39. 39.

    Johnson SR, et al. Methods to elicit beliefs for Bayesian priors: a systematic review. J Clin Epidemiol. 2010;63(4):355–69.

    PubMed  Article  Google Scholar 

  40. 40.

    Kuhnert PM, Martin TG, Griffiths SP. A guide to eliciting and using expert knowledge in Bayesian ecological models. Ecol Lett. 2010;13:900–14.

    PubMed  Article  Google Scholar 

  41. 41.

    Mullin TM. Understanding and supporting the process of probabilistic estimation. Pittsburgh: Carnegie-Mellon University; 1986.

    Google Scholar 

  42. 42.

    Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci. 1963;9(3):458–67.

    Article  Google Scholar 

  43. 43.

    Clemen RT, Winkler RL. Aggregating probability distributions. In: Edwards W, Miles Jr RF, von Winterfeldt D, editors. Advances in decision analysis: from foundations to applications. Cambridge: Cambridge University Press; 2007.

    Google Scholar 

  44. 44.

    Ayyub B. Elicitation of expert opinions for uncertainty and risks. Boca Raton: CRC Press; 2001.

    Google Scholar 

  45. 45.

    Rohrbaugh J. Improving the quality of group judgment: social judgment analysis and the nominal group technique. Organ Behav Hum Perform. 1981;28(2):272–88.

    Article  Google Scholar 

  46. 46.

    Sullivan W, Payne K. The appropriate elicitation of expert opinion in economic models. Pharmacoeconomics. 2011;29(6):455–9.

    PubMed  Article  Google Scholar 

  47. 47.

    Keeney S, McKenna H, Hasson F. The Delphi technique in nursing and health research. New York: Wiley; 2010. p. 208.

    Google Scholar 

  48. 48.

    Myers DG, Lamm H. The polarizing effect of group discussion: the discovery that discussion tends to enhance the average prediscussion tendency has stimulated new insights about the nature of group influence. Am Sci. 1975;63(3):297–303.

    CAS  PubMed  Google Scholar 

  49. 49.

    Sniezek JA. Groups under uncertainty: an examination of confidence in group decision making. Organ Behav Hum Decis Process. 1992;52(1):124–55.

    Article  Google Scholar 

  50. 50.

    White IR, Pocock SJ, Wang D. Eliciting and using expert opinions about influence of patient characteristics on treatment effects: a Bayesian analysis of the CHARM trials. Stat Med. 2005;24(24):3805–21.

    PubMed  Article  Google Scholar 

  51. 51.

    Shabaruddin F, Elliott R, Valle JW, Newman W, Payne K. Understanding chemotherapy treatment pathways of advanced colorectal cancer patients to inform an economic evaluation in the United Kingdom. Br J Cancer. 2010;103(3):315–23.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  52. 52.

    Genest C, McConway KJ. Allocating the weights in the linear opinion pool. J Forecast. 1990;9(1):53–73.

    Article  Google Scholar 

  53. 53.

    Moatti M, et al. Modeling of experts’ divergent prior beliefs for a sequential phase III clinical trial. Clin Trials. 2013;10(4):505–14.

    PubMed  Article  Google Scholar 

  54. 54.

    Wilson KJ. An investigation of dependence in expert judgement studies with multiple experts. Int J Forecast. 2017;33(1):325–36.

    Article  Google Scholar 

  55. 55.

    Hoelzer K, et al. Structured expert elicitation about listeria monocytogenes cross-contamination in the environment of retail deli operations in the United States. Risk Anal. 2012;32(7):1139–56.

    PubMed  Article  Google Scholar 

  56. 56.

    Wallsten TS, Budescu DV. State of the art-encoding subjective probabilities: a psychological and psychometric review. Manag Sci. 1983;29(2):151–73.

    Article  Google Scholar 

  57. 57.

    Cooke RM. Experts in uncertainty. Oxford: Oxford University Press; 1991.

    Google Scholar 

  58. 58.

    Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. 1950;78(1):1–3.

    Article  Google Scholar 

  59. 59.

    Murphy AH. A new vector partition of the probability score. J Appl Meteorol. 1973;12(4):595–600.

    Article  Google Scholar 

  60. 60.

    Yates JF. Subjective probability analysis. In: Wright G, Ayton P, editors. Subjective probability. London: Wiley; 1994. p. 382–410.

    Google Scholar 

  61. 61.

    Remus W, O’Conner M, Griggs K. Does feedback improve the accuracy of recurrent judgmental forecasts? Organ Behav Hum Decis Process. 1996;66(1):22–30.

    Article  Google Scholar 

  62. 62.

    Subbotin V. Outcome feedback effects on under- and overconfident judgments (general knowledge tasks). Organ Behav Hum Decis Process. 1996;66(3):268–76.

    Article  Google Scholar 

  63. 63.

    Bolger F, Rowe G. The aggregation of expert judgment: do good things come to those who weight? Risk Anal. 2015;35(1):12–5.

    Article  Google Scholar 

  64. 64.

    Chaloner K, et al. Graphical elicitation of a prior distribution for a clinical trial. Statistician. 1993;42:341–53.

    Article  Google Scholar 

  65. 65.

    National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. London: National Institute for Health and Care Excellence; 2013.

    Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Laura Bojke.

Ethics declarations


No funding was received for the work carried out in the preparation of this manuscript. Laura Bojke was supported in the preparation/submission of this paper by the HEOM Theme of the National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care Yorkshire and Humber (NIHR CLAHRC YH; The views and opinions expressed are those of the authors and not necessarily those of the National Health Service, the National Institute for Health Research, or the Department of Health.

Conflict of interest

Laura Bojke, Bogdan Grigore, Dina Jankovic, Jaime Peters, Marta Soares and Ken Stein have no conflicts of interest to declare.

Author contributions

LB, BG and DJ were primarily responsible for drafting the manuscript. JP, MS and KS contributed towards writing of the manuscript and commented on its various versions.



See Boxes 1, 2 and 3.

Box 1: Uses of elicitation in cost-effectiveness modelling

  • Generating an appropriate set of comparators.

  • Identifying appropriate patient pathways and relevant events.

  • Describing parameters and their associated uncertainty.

  • Quantifying the extent of bias, or improving generalisability from one context to another.

  • Characterising structural uncertainties either through generating differential weights for scenarios or by eliciting distributions of parameterized uncertainties.

  • Validating or calibrating model estimates.

Box 2: Application of the histogram method


Box 3: Examples of biases in elicitation

Biases in elicitation can include:

  • Biases associated with experts.

    • Motivation biases, e.g. when experts have an incentive (for example, financial) to reach a certain conclusion.

    • Cognitive biases: these commonly involve the use of heuristics, or ‘rules of thumb’, to help reach decisions, solve problems or form judgements quickly. Examples are:

      • Conjunction fallacy: When the probability of conjunction (combined) events is judged to be more likely than either of its constituents.

      • Availability: Where easy-to-recall events (such as natural disasters) are judged to have a high probability of occurring.

      • Hindsight bias: The tendency to overestimate the predictability of past events.

      • Anchoring effect: The tendency to rely on an anchor value that does not provide any information about the actual value.

  • Biases associated with elicitation methods

    • Structuring elicitation questions: Biases may arise from how the question is framed, for example if relevant events have been omitted, experts are less likely to consider these in replying. But biases can also occur when scales are used; for example, contraction bias occurs when the full range of a scale has not been presented to the expert.

    • Elicitation medium (e.g. interview or email survey) or aggregation method: As an example, experts in group meetings (typically conducted when consensus aggregation methods are applied) tend to adopt a stronger position, often resulting in overconfident statements.

    • Fitting probability distributions: The encoding of the summaries elicited from the expert as a distribution usually implies assumptions referring to the shape of the distribution that may differ from what the expert intended. For example an expert, when fitting distributions to his/her own beliefs, may be driven by familiar probability distribution shapes, in particular bell-shaped curves (familiarity bias).

      While the literature suggests that biases cannot be completely avoided, it is good practice to be aware of possible biases and to employ strategies to mitigate against these (debiasing) in both designing and conducting the elicitation exercise.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Bojke, L., Grigore, B., Jankovic, D. et al. Informing Reimbursement Decisions Using Cost-Effectiveness Modelling: A Guide to the Process of Generating Elicited Priors to Capture Model Uncertainties. PharmacoEconomics 35, 867–877 (2017).

Download citation