Skip to main content
SpringerLink
Log in

SHELF: The Sheffield Elicitation Framework

  • Chapter
  • First Online:
Elicitation

Part of the book series: International Series in Operations Research & Management Science ((ISOR,volume 261))

Abstract

The Sheffield elicitation framework is an expert knowledge elicitation framework that has been devised over a number of years and many substantial expert knowledge elicitation exercises to give a transparent and reliable way of collecting expert opinions. The framework is based on the principles of behavioural aggregation where a facilitator-guided group interact and share information to arrive at a consensus. It was originally designed for helping to elicit judgements about single uncertain variables, but, in recent years, the framework and the associated software implementations have been extended to accommodate judgements about more complex multidimensional variables and geographically-dispersed experts. In this chapter, we discuss the aims and foundations of the framework, its extensions and its notable applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 229.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 299.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 299.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Anson R, Bostrom R, Wynne B (1995) An experiment assessing group support system and facilitator effects on meeting outcomes. Manag Sci 41:189–208

    Article  Google Scholar 

  • Ayyub BM (2001) Elicitation of expert opinions for uncertainty and risks. CRC Press, Boca Raton

    Book  Google Scholar 

  • Bastin L, Williams M, Gosling JP, Truong P, Cornford D, Heuvelink G, Achard F (2011) Web based expert elicitation of uncertainties in environmental model inputs. In: Abstracts of the European Geosciences Union General Assembly 2011

    Google Scholar 

  • Bolger F (2018) The selection of experts for (probabilistic) expert knowledge elicitation. In: Dias LC, Morton A, Quigley J (eds) Elicitation: the science and art of structuring judgment. Springer, New York

    Google Scholar 

  • Brown RV, Lindley DV (1982) Improving judgment by reconciling incoherence. Theory Decis 14:113–32

    Article  Google Scholar 

  • Centre for Workforce Intelligence (2015) Elicitation methods: applying elicitation methods to robust workforce planning. http://www.cfwi.org.uk/publications/elicitation-methods-applying-elicitation-methods-to-robust-workforce-planning. Cited 15 Sept 2016

  • Clemen RT, Reilly T (1999) Correlations and copulas for decision and risk analysis. Manag Sci 45:208–24

    Article  Google Scholar 

  • Dalkey N (1969) An experimental study of group opinion: the Delphi method. Futures 1:408–426

    Article  Google Scholar 

  • Defence Science and Technology Laboratory (2015) The probabilistic elicitation of subjective data. https://www.gov.uk/government/publications/the-probabilistic-elicitation-of-subjective-data. Cited 15 Sept 2016

  • Dickey JM (1983) Multiple hypergeometric functions: probabilistic interpretations and statistical uses. J Am Stat Assoc 78:628–37

    Article  Google Scholar 

  • European Food Safety Authority (2014) Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA J 12(6):3734

    Article  Google Scholar 

  • French S (1985) Group consensus probability distributions: a critical survey. In: Bernardo JM et al. (eds) Bayesian statistics 2. Oxford University Press, Oxford, pp 183–202

    Google Scholar 

  • French S (2007) Web-enabled strategic GDSS, e-democracy and Arrow’s theorem: a Bayesian perspective. Decis Support Syst 43:1476–1484

    Article  Google Scholar 

  • Garthwaite PH, Kadane JB, O’Hagan A (2005) Statistical methods for eliciting probability distributions. J Am Stat Assoc 100:680–701

    Article  Google Scholar 

  • Girling AJ, Freeman G, Gordon JP, Poole-Wilson P, Scott DA, Lilford RJ (2007) Modeling payback from research into the efficacy of left-ventricular assist devices as destination therapy. Int J Technol Assess Health Care 23:269–77

    Article  Google Scholar 

  • Gosling JP, Oakley J, O’Hagan A (2007) Nonparametric elicitation for heavy-tailed prior distributions. Bayesian Anal 2:693–718

    Article  Google Scholar 

  • Gosling JP, Hart A, Mouat D, Sabirovic M, Scanlon S, Simmons A (2012) Quantifying experts’ uncertainty about the future cost of exotic diseases. Risk Anal 32:881–893

    Article  Google Scholar 

  • Gosling JP, Hart A, Owen H, Davies M, Li J, MacKay C (2013) A Bayes linear approach to weight-of-evidence risk assessment for skin allergy. Bayesian Anal 8:169–186

    Article  Google Scholar 

  • Higgins H, Dryden I, Green M (2012) A Bayesian elicitation of veterinary beliefs regarding systemic dry cow therapy: variation and importance for clinical trial design. Prev Vet Med 106:87–96

    Article  Google Scholar 

  • HM Treasury (2015) The Aqua Book: guidance on producing quality analysis for government. https://www.gov.uk/government/publications/the-aqua-book-guidance-on-producing-quality-analysis-for-government. Cited 15 Sept 2016

  • Iglesias C, Thompson A, Rogowski W, Payne K (2016) Reporting guidelines for the use of expert judgement in model-based economic evaluations. PharmacoEconomics 34(11):1161–1172

    Article  Google Scholar 

  • Kadane JB (1986) Progress toward a more ethical method for clinical trials. J Med Philos 11:85–404

    Article  Google Scholar 

  • Kadane JB, Wolfson L (1998). Experiences in elicitation. Statistician 47:3–19

    Google Scholar 

  • Kennedy M, Anderson C, O’Hagan A, Lomas M, Woodward F, Gosling JP, Heinemeyer A (2008) Quantifying uncertainty in the biospheric carbon flux for England and Wales. J R Stat Soc Ser A 11:109–135

    Google Scholar 

  • Kennedy M, Clough H, Turner J (2009) Case studies in Bayesian microbial risk assessments. Environ Health 8:S19

    Article  Google Scholar 

  • Kinnersley N, Day S (2013) Structured approach to the elicitation of expert beliefs for a Bayesian-designed clinical trial: a case study. Pharm Stat 12:104–113

    Article  Google Scholar 

  • Lark R, Lawley R, Barron A, Aldiss D, Ambrose K, Cooper A, Lee J, Waters C (2015) Uncertainty in mapped geological boundaries held by a national geological survey: eliciting the geologists’ tacit error model. Solid Earth 6:727–745

    Article  Google Scholar 

  • Lee L, Pringle K, Reddington C, Mann G, Stier P, Spracklen D, Pierce J, Carslaw K (2013) The magnitude and causes of uncertainty in global model simulations of cloud condensation nuclei. Atmos Chem Phys 13:8879–8914

    Article  Google Scholar 

  • Lindley DV, Tversky A, Brown RV (1979) On the reconciliation of probability assessments. J R Stat Soc Ser A 1:146–80

    Article  Google Scholar 

  • Linstone HA, Turoff M (1975) The Delphi method: techniques and applications. Addison-Wesley, Boston

    Google Scholar 

  • Meads C, Auguste P, Davenport C, MaÅ‚ysiak S, Sundar S, Kowalska M, Zapalska A, Guest P, Thangaratinam S, Martin-Hirsch P et al (2013) Positron emission tomography/computerised tomography imaging in detecting and managing recurrent cervical cancer: systematic review of evidence, elicitation of subjective probabilities and economic modelling. Health Technol Assess 17:1–323

    Google Scholar 

  • Morgan M, Henrion M (1990) Uncertainty: a guide to dealing with uncertainty in quantitative risk and policy analysis. Cambridge University Press, New York

    Book  Google Scholar 

  • Morris D, Oakley J, Crowe J (2014) A web-based tool for eliciting probability distributions from experts. Environ Model Softw 52:1–4

    Article  Google Scholar 

  • Myers DG, Lamm H (1975) The polarizing effect of group discussion. Am Sci 63:297–303

    Google Scholar 

  • Oakley J, O’Hagan A (2002) Bayesian inference for the uncertainty distribution of computer model outputs. Biometrika 89:769–84

    Article  Google Scholar 

  • Oakley JE, O’Hagan A (2007) Uncertainty in prior elicitations: a nonparametric approach. Biometrika 94:427–41

    Article  Google Scholar 

  • Oakley J, O’Hagan A (2014) SHELF: the Sheffield elicitation framework (version 2.0). http://www.tonyohagan.co.uk/shelf/. Accessed 7 Sept 2016

  • O’Hagan A (1988) Probability: methods and measurement. Chapman and Hall, London

    Book  Google Scholar 

  • O’Hagan A (1998) Eliciting expert beliefs in substantial practical applications. Statistician 47:21–35

    Google Scholar 

  • O’Hagan A, Buck CE, Daneshkhah A, Eiser JR, Garthwaite PH, Jenkinson DJ, Oakley JE, Rakow T (2006) Uncertain judgements: eliciting experts’ probabilities. Wiley, Chichester

    Book  Google Scholar 

  • Raiffa H (1968). Decision analysis: introductory lectures on choices under uncertainty. Addison-Wesley, Reading

    Google Scholar 

  • R Core Team (2016) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna https://www.R-project.org/. Accessed 7 Sept 2016

  • Ren S, Oakley J (2014) Assurance calculations for planning clinical trials with time-to-event outcomes. Stat Med 33:31–45

    Article  Google Scholar 

  • Rowe G, Wright G (2001) Expert opinions in forecasting: the role of the Delphi technique. In: Armstrong JS (ed) Principles of forecasting. Springer, New York, p 125–144

    Chapter  Google Scholar 

  • Scholten L, Scheidegger A, Reichert P, Maurer M (2013) Combining expert knowledge and local data for improved service life modeling of water supply networks. Environ Model Softw 42:1–16

    Article  Google Scholar 

  • Sperber D, Mortimer D, Lorgelly P, Berlowitz D (2013) An expert on every street corner? Methods for eliciting distributions in geographically dispersed opinion pools. Value Health 16:434–437

    Article  Google Scholar 

  • Stone M (1961) The opinion pool. Ann Math Stat 32:1339–1342

    Article  Google Scholar 

  • Thomas EA, Ross BH (1980) On appropriate procedures for combining probability distributions within the same family. J Math Psychol 21:136–52

    Article  Google Scholar 

  • Truong P, Heuvelink G, Gosling JP (2013) Web-based tool for expert elicitation of the variogram. Comput Geosci 51:390–399

    Article  Google Scholar 

  • Usher W, Strachan N (2013) An expert elicitation of climate, energy and economic uncertainties. Energy policy 61:811–821

    Article  Google Scholar 

  • Winkler RL (1967) The quantification of judgment: Some methodological suggestions. J Am Stat Assoc 62:1105–1120

    Article  Google Scholar 

  • Zapata-Vázquez R, O’Hagan A, Soares Bastos L (2014) Eliciting expert judgements about a set of proportions. J Appl Stat 41:1919–1933

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Mathematics, University of Leeds, Leeds, LS2 9JT, UK

    John Paul Gosling

Authors
  1. John Paul Gosling
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to John Paul Gosling .

Editor information

Editors and Affiliations

  1. Faculty of Economics, CeBER and INESC Coimbra, University of Coimbra, Coimbra, Portugal

    Luis C. Dias

  2. Department of Management Science, University of Strathclyde, Scotland, United Kingdom

    Alec Morton

  3. Department of Management Science, University of Strathclyde, Scotland, United Kingdom

    John Quigley

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gosling, J.P. (2018). SHELF: The Sheffield Elicitation Framework. In: Dias, L., Morton, A., Quigley, J. (eds) Elicitation. International Series in Operations Research & Management Science, vol 261. Springer, Cham. https://doi.org/10.1007/978-3-319-65052-4_4

Download citation

Publish with us

Policies and ethics

Search

Navigation