Skip to main content
View expanded cover

Elicitation pp 95–117Cite as

IDEA for Uncertainty Quantification

Part of the International Series in Operations Research & Management Science book series (ISOR,volume 261)

Abstract

It is generally agreed that an elicitation protocol for quantifying uncertainty will always benefit from the involvement of more than one domain expert. The two key mechanisms by which judgements may be pooled across experts are through striving for consensus, via behavioural aggregation, where experts share and discuss information, and via mathematical methods, where judgements are combined using a mechanistic rule. Mixed approaches combine elements of both deliberative (behavioural) and mechanical (mathematical) styles of aggregation.

This chapter outlines a mixed-aggregation protocol called IDEA. It synthesises specific elements from several of the classical structured expert judgement approaches. IDEA encourages experts to Investigate, Discuss, and Estimate, and concludes with a mathematical Aggregation of judgements.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-65052-4_5
  • Chapter length: 23 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   229.00
Price excludes VAT (USA)
  • ISBN: 978-3-319-65052-4
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   299.99
Price excludes VAT (USA)
Hardcover Book
USD   299.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 5.1
Fig. 5.2
Fig. 5.3
Fig. 5.4
Fig. 5.5
Fig. 5.6

Notes

  1. 1.

    However, where a group consensus judgement cannot be reached, individual expert distributions can be elicited and combined using a mathematical aggregation technique. Or alternatively, where consensus is not the aim, the resulting spread of expert viewpoints following discussion can be maintained and presented to decision-makers (Morgan 2015).

  2. 2.

    The best estimate may be also interpreted as the mode of the distribution. Methods for building a distribution that complies with the mode and two specified quantiles are proposed in Salomon (2013). However the interpretation of the best estimate and its use in constructing a distribution should be clearly specified prior to the elicitation.

  3. 3.

    http://www.iarpa.gov/index.php/research-programs/ace.

  4. 4.

    http://intelgame.acera.unimelb.edu.au/.

  5. 5.

    Performance was measured using the average Brier score. This measure was imposed by the forecasting tournament rules and all participating team had to use it.

  6. 6.

    The calibration measure for events is based on similar concepts as the ones presented in Chap. 2 of this book (Quigley et al. 2018), when the calibration score is described for evaluating assessments about continuous variables.

  7. 7.

    The relative information is usually known as the Kullback–Leibler divergence, or information divergence, or information gain, or relative entropy.

  8. 8.

    Three quarters of the Brier scores and the average confidence scores are better in the second round, and two thirds of the calibration scores and the informativeness scores are better in the second round.

References

  • Aspinall W (2010) A route to more tractable expert advice. Nature 463:294–295

    CrossRef  Google Scholar 

  • Aspinall W, Cooke R (2013) Quantifying scientific uncertainty from expert judgement elicitation. In: Rougier J, Sparks S, Hill L (eds) Risk and uncertainty assessment for natural hazards, chap 10. Cambridge University Press, Cambridge, pp 64–99

    Google Scholar 

  • Barons M, Wright S, Smith J (2018) Eliciting probabilistic judgements for integrating decision support systems. In: Dias L, Morton A, Quigley J (eds) Elicitation: the science and art of structuring judgment, chap 17. Springer, New York

    Google Scholar 

  • Bolger F (2018) The selection of experts for (probabilistic) expert knowledge elicitation. In: Dias L, Morton A, Quigley J (eds) Elicitation: the science and art of structuring judgment, chap 16. Springer, New York

    Google Scholar 

  • Bolger F, Stranieri A, Wright G, Yearwood J (2011) Does the Delphi process lead to increased accuracy in group-based judgmental forecasts or does it simply induce consensus amongst judgmental forecasters? Technol Forecast Soc Chang 78(9):1671–1680

    CrossRef  Google Scholar 

  • Bolger F, Hanea A, O ́Hagan A, Mosbach-Schulz O, Oakley J, Rowe G, Wenholt M (2014) Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA J 12(6):3745

    Google Scholar 

  • Booker JM, Meyer M (1987) Sources of correlation between experts: empirical results from two extremes. Los Alamos National Lab., NM (USA); Nuclear Regulatory Commission, Washington, DC (USA). Office of Nuclear Regulatory Research

    Google Scholar 

  • Brier G (1950) Verification of forecasts expressed in terms of probability. Mon Weather Rev 78:1–3

    CrossRef  Google Scholar 

  • Burgman M (2016) Trusting judgements: how to get the best out of experts. Cambridge University Press, Cambridge

    CrossRef  Google Scholar 

  • Burgman M, Carr A, Godden L, Gregory R, McBride M, Flander L, Maguire L (2011) Redefining expertise and improving ecological judgment. Conserv Lett 4:81–87. doi:10.1111/j.1755-263X.2011.00165.x

    CrossRef  Google Scholar 

  • Clemen R, Winkler R (1999) Combining probability distributions from experts in risk analysis. Risk Anal 19:187–203

    Google Scholar 

  • Cooke R (1991) Experts in uncertainty: opinion and subjective probability in science. Environmental ethics and science policy series. Oxford University Press, Oxford

    Google Scholar 

  • Cooke R, Goossens L (2008) TU Delft expert judgment data base. Reliab Eng Syst Saf 93(5): 657–674

    CrossRef  Google Scholar 

  • Dalkey N (1969) An experimental study of group opinion: the Delphi method. Futures 1:408–426

    CrossRef  Google Scholar 

  • de Finetti B (1962) Does it make sense to speak of ‘good probability appraisers’? In: Good, J. (ed) The scientist speculates: an anthology of partly baked ideas. Basic Books, New York, pp 357–363

    Google Scholar 

  • Gosling, J (2018) SHELF: the Sheffield elicitation framework. In: Dias L, Morton A, Quigley J (eds) Elicitation: the science and art of structuring judgment, chap 4. Springer, New York

    Google Scholar 

  • Hanea A, Burgman M, McBride M, Wintle B (2017) The Value of performance weights and discussion in aggregated expert judgements. Risk Anal (re-submitted June 2017)

    Google Scholar 

  • Hanea A, McBride M, Burgman M, Wintle B (2016) Classical meets modern in the IDEA protocol for structured expert judgement. J Risk Res doi:10.1080/13669877.2016.1215346 (Available online 9 Aug)

    Google Scholar 

  • Hanea A, McBride M, Burgman M, Wintle B, Fidler F, Flander L, Manning B, Mascaro S (2016) Investigate discuss estimate aggregate for structured expert judgement. Int J Forecast doi:10.1080/13669877.2016.1215346 (Available online 8 June)

    Google Scholar 

  • Hemming V, Burgman M, Hanea A, McBride M, Wintle B (2017) Preparing and implementing a structured expert elicitation using the IDEA protocol. Methods Ecol Evol, Accepted on 20.07.2017

    Google Scholar 

  • Hoel P (1971) Introduction to mathematical statistics. Wiley, New York

    Google Scholar 

  • Kerr N, Tindale R (2011) Group-based forecasting?: a social psychological analysis. Int J Forecast 27:14–40

    CrossRef  Google Scholar 

  • McBride M, Garnett S, Szabo J, Burbidge A, Butchart S, Christidis L, Dutson G, Ford H, Loyn R, Watson DM, Burgman M (2012) Structured elicitation of expert judgments for threatened species assessment: a case study on a continental scale using email. Methods Ecol. Evol. 3: 906–920

    CrossRef  Google Scholar 

  • Mellers B, Ungar L, Baron J, Ramos J, Gurcay B, Fincher K, Tetlock P (2014) Psychological strategies for winning a geopolitical forecasting tournament. Psychol Sci 25(4):1106–1115

    CrossRef  Google Scholar 

  • Mellers B, Stone E, Atanasov P, Rohrbaugh N, Metz S, Ungar L, Bishop M, Horowitz M, Merkle E, Tetlock P (2015) The psychology of intelligence analysis: drivers of prediction accuracy in world politics. J Exp Psychol Appl 21:1–14

    CrossRef  Google Scholar 

  • Morgan MG (2015) Our knowledge of the world is often not simple: policymakers should not duck that fact, but should deal with it. Risk Anal 35:19–20. doi:10.1111/risa.12306

    CrossRef  Google Scholar 

  • Murphy A (1973) A new vector partition of the probability score. J Appl Meteorol 12(4):595–600

    CrossRef  Google Scholar 

  • Murphy M, Black N, Lamping D, Mckee C, Sanderson C (1998) Consensus development methods and their use in clinical guideline development. Health Technol Assess 2(3):1–88

    Google Scholar 

  • O’Hagan A, Buck C, Daneshkhah A, Eiser J, Garthwaite P, Jenkinson D, Oakley J, Rakow T (2006) Uncertain judgements: eliciting experts’ probabilities. Wiley, London

    CrossRef  Google Scholar 

  • Quigley J, Colson A, Aspinall W, Cooke R (2018) Elicitation in the classical method. In Dias L, Morton A, Quigley J (eds) Elicitation: the science and art of structuring judgment, chap 2. Springer, New York

    Google Scholar 

  • Rowe G, Wright G (2001) Expert opinions in forecasting: the role of the Delphi technique. In: Principles of forecasting: a handbook for researchers and practitioners. Kluwer Academic Publishers, Norwell, pp. 125–144

    CrossRef  Google Scholar 

  • Salomon Y (2013) Unimodal density estimation with applications in expert elicitation and decision making under uncertainty. Ph.D. thesis, Department of Mathematics and Statistics, The University of Melbourne

    Google Scholar 

  • Savage L (1971) Elicitation of personal probabilities and expectations. J Am Stat Assoc 66: 783–801

    CrossRef  Google Scholar 

  • Soll J, Klayman J (2004) Overconfidence in interval estimates. J Exp Psychol Learn Mem Cogn 30:299–314

    CrossRef  Google Scholar 

  • Speirs-Bridge A, Fidler F, McBride M, Flander L, Cumming G, Burgman M (2010) Reducing overconfidence in the interval judgments of experts. Risk Anal 30:512–523

    CrossRef  Google Scholar 

  • Ungar L, Mellers B, Satopaa V, Baron J, Tetlock P, Ramos J, Swift S (2012) The good judgment project: a large scale test of different methods of combining expert predictions. In: AAAI fall symposium series. (AAAI Technical Report FS-12-06)

    Google Scholar 

  • Valverde L (2001) Expert judgment resolution in technically-intensive policy disputes. In: Assessment and management of environmental risks. Kluwer Academic Publishers, Norwell, pp 221–238

    CrossRef  Google Scholar 

  • von der Gracht H (2012) Consensus measurement in Delphi studies: review and implications for future quality assurance. Tech Forcasting Soc Chang 79:1525–1536

    CrossRef  Google Scholar 

  • Wilson K, Farrow M Combining judgements from correlated experts. In: Dias L, Morton A, Quigley J (eds) Elicitation: the science and art of structuring judgment, chap 9. Springer, New York (2018)

    Google Scholar 

  • Winkler R, Jose V (2010) Scoring rules. In: Wiley encyclopedia of operations research and management science. Wiley, New York

    Google Scholar 

  • Winkler R, Murphy A (1968) “Good” probability assessors. J Appl Meteorol 7:751–758

    CrossRef  Google Scholar 

  • Wintle B, Mascaro M, Fidler F, McBride M, Burgman M, Flander L, Saw G, Twardy C, Lyon A, Manning B (2012) The intelligence game: assessing Delphi groups and structured question formats. In: Proceedings of the 5th Australian security and intelligence conference

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CEBRA, University of Melbourne, Parkville, VIC, Australia

    Anca M. Hanea & Victoria Hemming

  2. Centre for Environmental Policy, Imperial College London, London, UK

    Mark Burgman

Authors
  1. Anca M. Hanea
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Burgman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Victoria Hemming
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anca M. Hanea .

Editor information

Editors and Affiliations

  1. Faculty of Economics, CeBER and INESC Coimbra, University of Coimbra, Coimbra, Portugal

    Luis C. Dias

  2. Department of Management Science, University of Strathclyde, Scotland, United Kingdom

    Alec Morton

  3. Department of Management Science, University of Strathclyde, Scotland, United Kingdom

    John Quigley

Rights and permissions

Reprints and Permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Verify currency and authenticity via CrossMark

Cite this chapter

Hanea, A.M., Burgman, M., Hemming, V. (2018). IDEA for Uncertainty Quantification. In: Dias, L., Morton, A., Quigley, J. (eds) Elicitation. International Series in Operations Research & Management Science, vol 261. Springer, Cham. https://doi.org/10.1007/978-3-319-65052-4_5

Download citation