, Volume 31, Issue 11, pp 991–1003 | Cite as

Methods to Elicit Probability Distributions from Experts: A Systematic Review of Reported Practice in Health Technology Assessment

  • Bogdan Grigore
  • Jaime Peters
  • Christopher Hyde
  • Ken Stein
Systematic Review



Elicitation is a technique that can be used to obtain probability distribution from experts about unknown quantities. We conducted a methodology review of reports where probability distributions had been elicited from experts to be used in model-based health technology assessments.


Databases including MEDLINE, EMBASE and the CRD database were searched from inception to April 2013. Reference lists were checked and citation mapping was also used. Studies describing their approach to the elicitation of probability distributions were included. Data was abstracted on pre-defined aspects of the elicitation technique. Reports were critically appraised on their consideration of the validity, reliability and feasibility of the elicitation exercise.


Fourteen articles were included. Across these studies, the most marked features were heterogeneity in elicitation approach and failure to report key aspects of the elicitation method. The most frequently used approaches to elicitation were the histogram technique and the bisection method. Only three papers explicitly considered the validity, reliability and feasibility of the elicitation exercises.


Judged by the studies identified in the review, reports of expert elicitation are insufficient in detail and this impacts on the perceived usability of expert-elicited probability distributions. In this context, the wider credibility of elicitation will only be improved by better reporting and greater standardisation of approach. Until then, the advantage of eliciting probability distributions from experts may be lost.


Health Technology Assessment Bisection Method Elicitation Method Elicitation Process Expert Elicitation 



This research was funded by the National Institute of Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (CLAHRC) for the South West Peninsula. The views expressed in this publication are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

Author Contributions

All authors contributed to the conception of the study. BG and JP collected and extracted the data. All authors contributed to interpretation of the data and to the final manuscript. Ken Stein is the guarantor for the overall content of this paper.

Conflict of Interest

All authors have no conflicts of interest.

Supplementary material

40273_2013_92_MOESM1_ESM.doc (23 kb)
Supplementary material (DOC 23 kb)


  1. 1.
    Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J Clin Nurs. 2003;12(1):77–84.PubMedCrossRefGoogle Scholar
  2. 2.
    Evans C, Crawford B. Expert judgement in pharmacoeconomic studies: guidance and future use. Pharmacoeconomics. 2000;17(6):545–53.PubMedCrossRefGoogle Scholar
  3. 3.
    Sullivan W, Payne K. The appropriate elicitation of expert opinion in economic models: making expert data fit for purpose. Pharmacoeconomics. 2011;29(6):455–9.PubMedCrossRefGoogle Scholar
  4. 4.
    O’Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, et al. Uncertain judgements: eliciting experts’ probabilities. Chichester: Wiley; 2006.CrossRefGoogle Scholar
  5. 5.
    Claxton K, Sculpher M, McCabe C, Briggs A, Akehurst R, Buxton M, et al. Probabilistic sensitivity analysis for NICE technology assessment: not an optional extra. Health Econ. 2005;14(4):339–47.PubMedCrossRefGoogle Scholar
  6. 6.
    Claxton KP, Sculpher MJ. Using value of information analysis to prioritise health research. Pharmacoeconomics. 2006;24(11):1055–68.PubMedCrossRefGoogle Scholar
  7. 7.
    Philips Z, Ginnelly L, Sculpher M, Claxton K, Golder S, Riemsma R, et al. Review of guidelines for good practice in decision-analytic modelling in health technology assessment. Health Technol Assess. 2004;8(36):iii–iv, ix–xi, 1–158Google Scholar
  8. 8.
    Winkler RL, Poses RM. Evaluating and combining physicians’ probabilities of survival in an intensive care unit. Manag Sci. 1993;39:1526–43.CrossRefGoogle Scholar
  9. 9.
    Knol AB, Slottje P, van der Sluijs JP, Lebret E. The use of expert elicitation in environmental health impact assessment: a seven step procedure. Environ Health Glob Access Sci Source. 2010;9:19.Google Scholar
  10. 10.
    Garthwaite PH, Kadane JB, O’Hagan A. Statistical methods for eliciting probability distributions. J Am Stat Assoc. 2005;100(470):680–701.CrossRefGoogle Scholar
  11. 11.
    Winkler RL. The assessment of prior distributions in Bayesian analysis. J Am Stat Assoc. 1967;62(319):776–800.CrossRefGoogle Scholar
  12. 12.
    Soares MO, Bojke L, Dumville J, Iglesias C, Cullum N, Claxton K. Methods to elicit experts’ beliefs over uncertain quantities: application to a cost effectiveness transition model of negative pressure wound therapy for severe pressure ulceration. Stat Med. 2011;30(19):2363–80.PubMedCrossRefGoogle Scholar
  13. 13.
    Van Noortwijk JM, Dekker A, Cooke RM, Mazzuchi TA. Expert judgment in maintenance optimization. IEEE Trans Reliab. 1992;41(3):427–32.CrossRefGoogle Scholar
  14. 14.
    Speight PM, Palmer S, Moles DR, Downer MC, Smith DH, Henriksson M, et al. The cost-effectiveness of screening for oral cancer in primary care. Health Technol Assess 2006;10(14):1–144, iii–ivGoogle Scholar
  15. 15.
    Phillips L, Wisbey S. The elicitation of judgemental probability distributions from groups of experts: a description of the methodology and records of seven formal elicitation sessions held in 1991 and 1992. Report nss/r282. Didcot, UK: Nirex UK; 1993Google Scholar
  16. 16.
    Clemen RT, Winkler RL. Aggregating probability distributions. In: Advances in decision analysis: from foundations to applications. USA: Cambridge University Press; 2007. p. 154–76.Google Scholar
  17. 17.
    Kadane J, Wolfson LJ. Experiences in elicitation. J R Stat Soc Ser D (Stat). 1998;47(1):3–19.CrossRefGoogle Scholar
  18. 18.
    Johnson SR, Tomlinson GA, Hawker GA, Granton JT, Feldman BM. Methods to elicit beliefs for Bayesian priors: a systematic review. J Clin Epidemiol. 2010;63(4):355–69.PubMedCrossRefGoogle Scholar
  19. 19.
    Kinnersley N, Day S. Structured approach to the elicitation of expert beliefs for a Bayesian‐designed clinical trial: a case study. Pharm Stat. 2013;12(2):104–13.PubMedCrossRefGoogle Scholar
  20. 20.
    Keszei AP, Novak M, Streiner DL. Introduction to health measurement scales. J Psychosom Res. 2010;68(4):319–23.PubMedCrossRefGoogle Scholar
  21. 21.
    Garthwaite PH, Chilcott JB, Jenkinson DJ, Tappenden P. Use of expert knowledge in evaluating costs and benefits of alternative service provisions: a case study. Int J Technol Assess Health Care. 2008;24(3):350–7.PubMedCrossRefGoogle Scholar
  22. 22.
    Cooke RM. Experts in uncertainty: opinion and subjective probability in science. USA: Oxford University Press; 1991.Google Scholar
  23. 23.
    Wolpert RL. Eliciting and combining subjective judgments about uncertainty. Int J Technol Assess Health Care. 1989;5(4):537–57.PubMedCrossRefGoogle Scholar
  24. 24.
    Brodtkorb TH. Cost-effectiveness analysis of health technologies when evidence is scarce. Linköping: Linkoping University; 2010.Google Scholar
  25. 25.
    Haakma W. Expert elicitation to populate early health economic models of medical diagnostic devices in development. Twente: University of Twente; 2011.Google Scholar
  26. 26.
    Leal J, Wordsworth S, Legood R, Blair E. Eliciting expert opinion for economic models: an applied example. Value Health. 2007;10(3):195–203.PubMedCrossRefGoogle Scholar
  27. 27.
    Bojke L, Claxton K, Bravo-Vergel Y, Sculpher M, Palmer S, Abrams K. Eliciting distributions to populate decision analytic models. Value Health. 2010;13(5):557–64.PubMedCrossRefGoogle Scholar
  28. 28.
    Sperber D, Mortimer D, Lorgelly P, Berlowitz D. An expert on every street corner? Methods for eliciting distributions in geographically dispersed opinion pools. Value Health. 2013;16(2):434–7.PubMedCrossRefGoogle Scholar
  29. 29.
    McKenna C, McDaid C, Suekarran S, Hawkins N, Claxton K, Light K, et al. Enhanced external counterpulsation for the treatment of stable angina and heart failure: a systematic review and economic evaluation (structured abstract). Health Technol Assess. 2009;13(24):1–90.PubMedGoogle Scholar
  30. 30.
    Stevenson MD, Oakley JE, Lloyd Jones M, Brennan A, Compston JE, McCloskey EV, et al. The cost-effectiveness of an RCT to establish whether 5 or 10 years of bisphosphonate treatment is the better duration for women with a prior fracture. Med Decis Mak. 2009;29(6):678–89.CrossRefGoogle Scholar
  31. 31.
    Girling AJ, Freeman G, Gordon JP, Poole-Wilson P, Scott DA, Lilford RJ. Modeling payback from research into the efficacy of left-ventricular assist devices as destination therapy (structured abstract). Int J Technol Assess Health Care. 2007;23(2):269–77.PubMedCrossRefGoogle Scholar
  32. 32.
    Meads C, Auguste P, Davenport C, Małysiak S, Sundar S, Kowalska M, et al. Positron emission tomography/computerised tomography imaging in detecting and managing recurrent cervical cancer: systematic review of evidence, elicitation of subjective probabilities and economic modelling. Health Technol Assess. 2013;17(12):1–323.PubMedGoogle Scholar
  33. 33.
    Colbourn T, Asseburg C, Bojke L, Philips Z, Claxton K, Ades A, et al. Prenatal screening and treatment strategies to prevent group B streptococcal and other bacterial infections in early infancy: cost-effectiveness and expected value of information analyses. Health Technol Assess. 2007;11(29):1–226, iii.Google Scholar
  34. 34.
    Stevenson MD, Oakley JE, Chick SE, Chalkidou K. The cost-effectiveness of surgical instrument management policies to reduce the risk of vCJD transmission to humans. J Oper Res Soc. 2008;60(4):506–18.CrossRefGoogle Scholar
  35. 35.
    O’Leary RA, Choy SL, Murray JV, Kynn M, Denham R, Martin TG, et al. Comparison of three expert elicitation methods for logistic regression on predicting the presence of the threatened brush-tailed rock-wallaby Petrogale penicillata. Environmetrics. 2009;20(4):379–98.CrossRefGoogle Scholar
  36. 36.
    Moher D, Schulz KF, Simera I, Altman DG. Guidance for developers of health research reporting guidelines. PLoS Med. 2010;7(2):e1000217.PubMedCrossRefGoogle Scholar
  37. 37.
    Mazur A II. Disputes between experts. Minerva. 1973;11(2):243–62.CrossRefGoogle Scholar
  38. 38.
    Tversky A, Kahneman D. The framing of decisions and the psychology of choice. Science. 1981;211(4481):453–8.PubMedCrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2013

Authors and Affiliations

  • Bogdan Grigore
    • 1
  • Jaime Peters
    • 1
  • Christopher Hyde
    • 1
  • Ken Stein
    • 1
  1. 1.Peninsula Technology Assessment Group (PenTAG), Institute of Health Research, University of Exeter Medical SchoolUniversity of ExeterExeterUK

Personalised recommendations