, Volume 35, Issue 9, pp 867–877 | Cite as

Informing Reimbursement Decisions Using Cost-Effectiveness Modelling: A Guide to the Process of Generating Elicited Priors to Capture Model Uncertainties

  • Laura BojkeEmail author
  • Bogdan Grigore
  • Dina Jankovic
  • Jaime Peters
  • Marta Soares
  • Ken Stein
Practical Application


In informing decisions, utilising health technology assessment (HTA), expert elicitation can provide valuable information, particularly where there is a less-developed evidence-base at the point of market access. In these circumstances, formal methods to elicit expert judgements are preferred to improve the accountability and transparency of the decision-making process, help reduce bias and the use of heuristics, and also provide a structure that allows uncertainty to be expressed. Expert elicitation is the process of transforming the subjective and implicit knowledge of experts into their quantifiable expressions. The use of expert elicitation in HTA is gaining momentum, and there is particular interest in its application to diagnostics, medical devices and complex interventions such as in public health or social care. Compared with the gathering of experimental evidence, elicitation constitutes a reasonably low-cost source of evidence. Given its inherent subject nature, the potential biases in elicited evidence cannot be ignored and, due to its infancy in HTA, there is little guidance to the analyst wishing to conduct a formal elicitation exercise. This article attempts to summarise the stages of designing and conducting an expert elicitation, drawing on key literature and examples, most of which are not in HTA. In addition, we critique their applicability to HTA, given its distinguishing features. There are a number of issues that the analyst should be mindful of, in particular the need to appropriately characterise the uncertainty associated with model inputs and the fact that there are often numerous parameters required, not all of which can be defined using the same quantities. This increases the need for the elicitation task to be as straightforward as possible for the expert to complete.


Compliance with Ethical Standards


No funding was received for the work carried out in the preparation of this manuscript. Laura Bojke was supported in the preparation/submission of this paper by the HEOM Theme of the National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care Yorkshire and Humber (NIHR CLAHRC YH; The views and opinions expressed are those of the authors and not necessarily those of the National Health Service, the National Institute for Health Research, or the Department of Health.

Conflict of interest

Laura Bojke, Bogdan Grigore, Dina Jankovic, Jaime Peters, Marta Soares and Ken Stein have no conflicts of interest to declare.

Author contributions

LB, BG and DJ were primarily responsible for drafting the manuscript. JP, MS and KS contributed towards writing of the manuscript and commented on its various versions.


  1. 1.
    Hunger T, et al. Using expert opinion in health technology assessment: a guideline review. Int J Technol Assessm Health Care. 2016;32(3):131–9.CrossRefGoogle Scholar
  2. 2.
    Grigore B, et al. Methods to elicit probability distributions from experts: a systematic review of reported practice in health technology assessment. Pharmacoeconomics. 2013;31(11):991–1003.PubMedCrossRefGoogle Scholar
  3. 3.
    Hadorn D, et al. Use of expert knowledge elicitation to estimate parameters in health economic decision models. Int J Technol Assess Health Care. 2014;30(4):461–8.PubMedCrossRefGoogle Scholar
  4. 4.
    Iglesias CP, et al. Reporting guidelines for the use of expert judgement in model-based economic evaluations. Pharmacoeconomics. 2016;34(11):1161–72.PubMedCrossRefGoogle Scholar
  5. 5.
    Hart A, O’Hagan A, Quigley J, Bolger F. Training course on steering an expert knowledge elicitation. Final report. EFSA Supporting publ. 2016;13(5):1009E. doi: 10.2903/sp.efsa.2016.EN-1009.Google Scholar
  6. 6.
    O’Hagan A, et al. Uncertain judgements: eliciting experts’ probabilities. New York: Wiley; 2006.CrossRefGoogle Scholar
  7. 7.
    Hora SC, Von Winterfeldt D. Nuclear waste and future societies: a look into the deep future. Technol Forecast Soc Change. 1997;56(2):155–70.CrossRefGoogle Scholar
  8. 8.
    Jenkinson D. The elicitation of probabilities: a review of the statistical literature. Bayesian elicitation of experts’ probabilities (BEEP). Sheffield: Sheffield University; 2005.Google Scholar
  9. 9.
    Soares MO, et al. Methods to elicit experts’ beliefs over uncertain quantities: application to a cost effectiveness transition model of negative pressure wound therapy for severe pressure ulceration. Stat Med. 2011;30(19):2363–80.PubMedCrossRefGoogle Scholar
  10. 10.
    Clemen RT, Winkler RL. Aggregating probability distributions. In: Edwards W, Miles Jr R, Von Winterfeldt D, editors. Advances in decision analysis: from foundations to applications. Cambridge: Cambridge University Press; 2007. p. 154–76.CrossRefGoogle Scholar
  11. 11.
    French S. Aggregating expert judgement. Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. Serie A. Matematicas. 2011;105(1):181–206.Google Scholar
  12. 12.
    Johnson SR, et al. Methods to elicit beliefs for Bayesian priors: a systematic review. J Clin Epidemiol. 2010;63(4):355–69.PubMedCrossRefGoogle Scholar
  13. 13.
    Kattan MW, et al. The wisdom of crowds of doctors: their average predictions outperform their individual ones. Med Dec Mak. 2016;36(4):536–40.CrossRefGoogle Scholar
  14. 14.
    Cooke RM, Goossens LHJ. Expert judgement elicitation for risk assessments of critical infrastructures. J Risk Res. 2004;7(6):643–56.CrossRefGoogle Scholar
  15. 15.
    Knol AB, et al. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environ Health Glob Access Sci Source. 2010;9:19.Google Scholar
  16. 16.
    Kadane J, Wolfson LJ. Experiences in elicitation. J R Stat Soc Ser D Stat. 1998;47(1):3–19.CrossRefGoogle Scholar
  17. 17.
    Renooij S, Witteman C. Talking probabilities: communicating probabilistic information with words and numbers. Int J Approx Reas. 1999;22(3):169–94.CrossRefGoogle Scholar
  18. 18.
    Bruine de Bruin W, et al. What number is “fifty-fifty”? Redistributing excessive 50% responses in elicited probabilities. Risk Anal. 2002;22(4):713–23.PubMedCrossRefGoogle Scholar
  19. 19.
    Claxton K, et al. Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra. Health Econ. 2005;14(4):339–47.PubMedCrossRefGoogle Scholar
  20. 20.
    Leal J, Wordsworth S, Legood R, Blair E. Eliciting expert opinion for economic models: an applied example. Value Health. 2007;10(3):195–203.PubMedCrossRefGoogle Scholar
  21. 21.
    Grigore B, et al. A comparison of two methods for expert elicitation in health technology assessments. BMC Med Res Methodol. 2016;16:85.PubMedPubMedCentralCrossRefGoogle Scholar
  22. 22.
    Pibouleau L, Chevret S. An internet-based method to elicit experts’ beliefs for Bayesian priors: a case study in intracranial stent evaluation. Int J Technol Assess Health Care. 2014;30(4):1–8.CrossRefGoogle Scholar
  23. 23.
    Bojke L, et al. Eliciting distributions to populate decision analytic models. Value Health. 2010;13(5):557–64.PubMedCrossRefGoogle Scholar
  24. 24.
    McKenna C, et al. Enhanced external counterpulsation for the treatment of stable angina and heart failure: a systematic review and economic evaluation. Health Technol Assess. 2009;13(24):1–90, iii–iv, ix–xi.PubMedCrossRefGoogle Scholar
  25. 25.
    Speight PM, et al. The cost-effectiveness of screening for oral cancer in primary care. Health Technol Assess. 2006;10(14):1–144, iii–iv.PubMedCrossRefGoogle Scholar
  26. 26.
    Van Noortwijk JM, et al. Expert judgment in maintenance optimization. IEEE Trans Reliab. 1992;41(3):427–32.CrossRefGoogle Scholar
  27. 27.
    Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Publ Health. 2005;27:281–91.CrossRefGoogle Scholar
  28. 28.
    Knol AB, et al. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environ Health. 2010;9:19.PubMedPubMedCentralCrossRefGoogle Scholar
  29. 29.
    Baker E, et al. Facing the experts: survey mode and expert elicitation. Fondazione Eni Enrico Mattei. Nora di Lavoro; 2014.Google Scholar
  30. 30.
    Morris DE, Oakley JE, Crowe JA. A web-based tool for eliciting probability distributions from experts. Environ Model Softw. 2014;52:1–4.CrossRefGoogle Scholar
  31. 31.
    Expert Judgement Network 2016. Accessed 13 Jun 2017.
  32. 32.
    Elfadaly FG, Garthwaite PH. Eliciting Dirichlet and Connor–Mosimann prior distributions for multinomial models. TEST. 2013;22(4):628–46.CrossRefGoogle Scholar
  33. 33.
    Garthwaite PH, et al. Use of expert knowledge in evaluating costs and benefits of alternative service provisions: a case study. Int J Technol Assess Health Care. 2008;24(3):350–7.PubMedCrossRefGoogle Scholar
  34. 34.
    WikiBooks. Cognitive science: an introduction/biases and reasoning heuristics. 2016. Accessed 3 May 2017.
  35. 35.
    Tversky A, Kahneman D. The framing of decisions and the psychology of choice. Science. 1981;4481:453–8.CrossRefGoogle Scholar
  36. 36.
    Garthwaite PH, Kadane JB, O’Hagan A. Statistical methods for eliciting probability distributions. J Am Stat Assoc. 2005;100(470):680–701.CrossRefGoogle Scholar
  37. 37.
    Montibeller G, von Winterfeldt D. Cognative and motivational biases in decison and risk analysis. Risk Anal. 2015;35(7):1230–51.PubMedCrossRefGoogle Scholar
  38. 38.
    Kynn M. The ‘heuristics and biases’ bias in expert elicitation. J R Stat Soc Ser A Stat Soc. 2008;171(1):239–64.Google Scholar
  39. 39.
    Johnson SR, et al. Methods to elicit beliefs for Bayesian priors: a systematic review. J Clin Epidemiol. 2010;63(4):355–69.PubMedCrossRefGoogle Scholar
  40. 40.
    Kuhnert PM, Martin TG, Griffiths SP. A guide to eliciting and using expert knowledge in Bayesian ecological models. Ecol Lett. 2010;13:900–14.PubMedCrossRefGoogle Scholar
  41. 41.
    Mullin TM. Understanding and supporting the process of probabilistic estimation. Pittsburgh: Carnegie-Mellon University; 1986.Google Scholar
  42. 42.
    Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci. 1963;9(3):458–67.CrossRefGoogle Scholar
  43. 43.
    Clemen RT, Winkler RL. Aggregating probability distributions. In: Edwards W, Miles Jr RF, von Winterfeldt D, editors. Advances in decision analysis: from foundations to applications. Cambridge: Cambridge University Press; 2007.Google Scholar
  44. 44.
    Ayyub B. Elicitation of expert opinions for uncertainty and risks. Boca Raton: CRC Press; 2001.CrossRefGoogle Scholar
  45. 45.
    Rohrbaugh J. Improving the quality of group judgment: social judgment analysis and the nominal group technique. Organ Behav Hum Perform. 1981;28(2):272–88.CrossRefGoogle Scholar
  46. 46.
    Sullivan W, Payne K. The appropriate elicitation of expert opinion in economic models. Pharmacoeconomics. 2011;29(6):455–9.PubMedCrossRefGoogle Scholar
  47. 47.
    Keeney S, McKenna H, Hasson F. The Delphi technique in nursing and health research. New York: Wiley; 2010. p. 208.Google Scholar
  48. 48.
    Myers DG, Lamm H. The polarizing effect of group discussion: the discovery that discussion tends to enhance the average prediscussion tendency has stimulated new insights about the nature of group influence. Am Sci. 1975;63(3):297–303.PubMedGoogle Scholar
  49. 49.
    Sniezek JA. Groups under uncertainty: an examination of confidence in group decision making. Organ Behav Hum Decis Process. 1992;52(1):124–55.CrossRefGoogle Scholar
  50. 50.
    White IR, Pocock SJ, Wang D. Eliciting and using expert opinions about influence of patient characteristics on treatment effects: a Bayesian analysis of the CHARM trials. Stat Med. 2005;24(24):3805–21.PubMedCrossRefGoogle Scholar
  51. 51.
    Shabaruddin F, Elliott R, Valle JW, Newman W, Payne K. Understanding chemotherapy treatment pathways of advanced colorectal cancer patients to inform an economic evaluation in the United Kingdom. Br J Cancer. 2010;103(3):315–23.PubMedPubMedCentralCrossRefGoogle Scholar
  52. 52.
    Genest C, McConway KJ. Allocating the weights in the linear opinion pool. J Forecast. 1990;9(1):53–73.CrossRefGoogle Scholar
  53. 53.
    Moatti M, et al. Modeling of experts’ divergent prior beliefs for a sequential phase III clinical trial. Clin Trials. 2013;10(4):505–14.PubMedCrossRefGoogle Scholar
  54. 54.
    Wilson KJ. An investigation of dependence in expert judgement studies with multiple experts. Int J Forecast. 2017;33(1):325–36.CrossRefGoogle Scholar
  55. 55.
    Hoelzer K, et al. Structured expert elicitation about listeria monocytogenes cross-contamination in the environment of retail deli operations in the United States. Risk Anal. 2012;32(7):1139–56.PubMedCrossRefGoogle Scholar
  56. 56.
    Wallsten TS, Budescu DV. State of the art-encoding subjective probabilities: a psychological and psychometric review. Manag Sci. 1983;29(2):151–73.CrossRefGoogle Scholar
  57. 57.
    Cooke RM. Experts in uncertainty. Oxford: Oxford University Press; 1991.Google Scholar
  58. 58.
    Brier GW. Verification of forecasts expressed in terms of probability. Mon Weather Rev. 1950;78(1):1–3.CrossRefGoogle Scholar
  59. 59.
    Murphy AH. A new vector partition of the probability score. J Appl Meteorol. 1973;12(4):595–600.CrossRefGoogle Scholar
  60. 60.
    Yates JF. Subjective probability analysis. In: Wright G, Ayton P, editors. Subjective probability. London: Wiley; 1994. p. 382–410.Google Scholar
  61. 61.
    Remus W, O’Conner M, Griggs K. Does feedback improve the accuracy of recurrent judgmental forecasts? Organ Behav Hum Decis Process. 1996;66(1):22–30.CrossRefGoogle Scholar
  62. 62.
    Subbotin V. Outcome feedback effects on under- and overconfident judgments (general knowledge tasks). Organ Behav Hum Decis Process. 1996;66(3):268–76.CrossRefGoogle Scholar
  63. 63.
    Bolger F, Rowe G. The aggregation of expert judgment: do good things come to those who weight? Risk Anal. 2015;35(1):12–5.CrossRefGoogle Scholar
  64. 64.
    Chaloner K, et al. Graphical elicitation of a prior distribution for a clinical trial. Statistician. 1993;42:341–53.CrossRefGoogle Scholar
  65. 65.
    National Institute for Health and Care Excellence. Guide to the methods of technology appraisal. London: National Institute for Health and Care Excellence; 2013.Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Centre for Health EconomicsUniversity of YorkYorkUK
  2. 2.Peninsula Technology Assessment GroupUniversity of ExeterExeterUK

Personalised recommendations