Skip to main content
SpringerLink
Log in

The Selection of Experts for (Probabilistic) Expert Knowledge Elicitation

  • Chapter
  • First Online:
Elicitation

Part of the book series: International Series in Operations Research & Management Science ((ISOR,volume 261))

Abstract

Several different EKE protocols are reviewed in this volume, each with their pros and cons, but any is only as good as the quality of the experts and their judgments. In this chapter a structured approach to the selection of experts for EKE is presented that is grounded in psychological research.

In Part I various definitions of expertise are considered, and indicators and measures that can be used for the selection of experts are identified. Next, some ways of making judgements of uncertain quantities are discussed, as are factors influencing judgment quality.

In Part II expert selection is considered within an overall policy-making process. Following the analysis of Part I, two new instruments are presented that can help guide the selection process: expert profiles provide structure to the initial search, while a questionnaire permits matching of experts to the profiles, and assessment of training needs. Issues of expert retention and documentation are also discussed.

It is concluded that although the analysis offered in this chapter constitutes a starting point there are many questions still to be answered to maximize EKE’s contribution. A promising direction is research that focusses on the interaction between experts and the tasks they perform.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 229.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 299.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 299.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    This is a manifestation of the ‘paradox of expertise’ (e.g. Dror 2011), which is that experts become worse in some respects as they become more expert e.g. less flexible, creative and responsive; more biased etc. This is because knowledge and reasoning become ‘fossilized’: less amenable to inspection, change and communication.

  2. 2.

    In the Delphi procedure, ‘groups’ of experts—who never meet or interact directly, and are anonymous to each other (all to reduce sources of social bias)—are polled for their opinions. These opinions are usually point estimates or forecasts of event occurrence (see e.g. Chap. 5 for a discussion of the elicitation and evaluation of such judgments) but can also be judgments of uncertain quantities expressed as probability intervals or distributions (see Bolger et al. 2014): reasons for judgments are also often elicited. Once experts have individually expressed their opinions they are collated by a facilitator and fed back to the expert panel (most normally quantitative estimates are averaged in some manner, and qualitative responses summarized, although individual responses may also be fed back if the group is not too large). The experts are then invited to revise their opinions in the light of the feedback and resubmit them to the facilitator. The process continues through a number of iterations usually until opinion change ceases. Normally the aggregated judgements from the final round are the output although partially aggregated or disaggregated judgements can be submitted if the process fails to lead to consensus.

  3. 3.

    Although potentially non-probabilistic modes of expressing uncertainty, such as natural-language terms, could be used these have not been found to be easily converted to the probabilities usually required for policy and decision-making (see e.g. Dhami and Wallsten 2005; Wallsten and Budescu 1995).

  4. 4.

    For instance, stock price movements have been characterized as random (e.g. Fama 1965). Although more recent research suggests that stock markets are, in fact, predictable in the long term (e.g. De Bondt and Thaler 1989) it is still agreed that it is not in the short-term, contrary to the beliefs of ‘day-traders’. It may often be the case that ‘experts’ believe there to be predictability where there is not, or it is rather low. In such situations, there can, of course, be little or no expertise (see e.g. forecasting of GDP growth, Budescu and Chen 2015) nor variation in performance. Further, perceived ability where there is none is another name for ‘overconfidence’ (more generally, insensitivity to task difficulty will lead to miscalibration).

  5. 5.

    Bolger and Rowe (2015a) identify a number of problems with this approach, including finding a sufficient number of suitable seeds—ones that draw on the same expert knowledge as the target. They also comment that this ‘Classical Method’? is atheoretical, in that it is not founded on any particular conceptualization of expert knowledge, and propose a cue-based approach that would provide a reasoned basis for the selection of similar seeds (i.e. those that are related by the cues used to judge them). This cue-based approach is outlined in Sections “Cue-Based Judgements” and 16.2.4.2 below.

  6. 6.

    However it must be stressed that this is just one study (which fails to report all the potentially relevant correlations). Further, we do not know the extent to which the tests of substantive expertise are good measures of actual expert performance on, for example, a real-world risk-assessment or forecasting task.

  7. 7.

    The other two traits being extroversion and neuroticism.

  8. 8.

    It is possible that the advantages of some personality traits are protocol-dependent. For example, conscientiousness might be good for a remotely administered elicitation such as is often the case in Delphi, while agreeableness might be particularly helpful in protocols that require face-to-face interaction. Openness to experience is probably a useful characteristic in both protocols as it should assist opinion change towards the true value.

  9. 9.

    Indeed, the ‘Theory of Errors’ (Dalkey 1975; Parenté and Anderson-Parenté 1987), which is the leading account for why the Delphi technique works, assumes those who stick are on average closer to the truth than those who shift—Bolger and Wright (2011) propose that in order to achieve ‘virtuous opinion change’—i.e. opinion change towards the truth—rationales for opinions should be fed back between Delphi rounds rather than confidence as the former will be better indicators that the expert is knowledgeable about the topic than the latter.

  10. 10.

    There is some empirical support for the suggestion that relative frequency is a more natural way of representing uncertainty than probability (Gigerenzer and Hoffrage 1995) thus posing questions as relative frequencies rather than probabilities might be an alternative to training.

  11. 11.

    In principle, many of these things could be done remotely but I am not aware of any existing protocols or software to support this.

  12. 12.

    However, Quigley and his colleagues use maps when eliciting priors with engineers assessing the reliability of new systems (e.g., Hodge et al. 2001; Walls et al. 2006—with the aerospace industry).

  13. 13.

    Support Theory also is well-developed with regard to uncertainty assessment but its primary focus is the assessment of the likelihood of categorical assessments (e.g. of event occurrences, correctness, or truth) rather than values of continuous variables.

  14. 14.

    Although as already indicated, there is still an ongoing debate as to the source and degree (and even existence) of overconfidence in such tasks—see Olsson 2014, for a recent review.

  15. 15.

    Bazerman and Moore (2008) suggest that experts also need to have coherent models (mental or formal) in order to make good quality (probability) judgements. So models may be added to learnability and ecological validity of tasks as a criterion for well-calibrated experts.

  16. 16.

    For example, ‘bootstrap’ linear models—models derived from regressing expert judgments onto the cues that they are presumed to use—make better predictions of a criterion than do the original unaided experts (e.g. Goldberg 1970) because they apply the judgment model more consistently.

  17. 17.

    In contrast to the ‘crowdsourcing’ approach mentioned above whereby a large number of people with little or no domain expertise are polled (see, e.g. Budescu and Chen 2015).

  18. 18.

    These experts will be referred to later as ‘super-experts’ since they have an overview of the problem as a whole and are responsible for recruitment, selection and management of any other experts used in the process.

  19. 19.

    This questionnaire was an adaptation of the one first developed by Wright et al. (2004), based on earlier work by Bolger and Wright (1994) and Rowe and Wright (2001): the example questions presented in Table 16.2 have been further edited by the current author since the 2014 version.

  20. 20.

    However, in some other important respects, expert selection is not like job selection i.e. you want to find people with the right skills rather than reject people with the wrong skills… so it is more akin to head hunting.

  21. 21.

    Potentially, scores for weighting could be derived from this questionnaire but this has not been attempted yet, let alone any validation of weights thus derived, therefore this is something for future research.

References

  • Alvarado-Valencia J, Barrero LH, Onkal D, Dennerlein JT (2017) Expertise, credibility of systems forecasts and integration of methods in judgmental demand forecasting. Int J Forecast 33(1):298–313

    Article  Google Scholar 

  • Anderson JR (1982) Acquisition of cognitive skill. Psychol Rev 89:369–406

    Article  Google Scholar 

  • Aspinall W (2010) A route to more tractable expert advice. Nature 463:294–295

    Article  Google Scholar 

  • Bargh JA (1994) The four horsemen of automaticity: intention, awareness, efficiency, and control as separate issues. In: Wyer R, Srull T (eds) Handbook of social cognition. Lawrence Erlbaum, Hillsdale, NJ, pp 1–40

    Google Scholar 

  • Barron G, Erev I (2003) Small feedback-based decisions and their limited correspondence to description-based decisions. J Behav Decis Mak 16(3):215–233

    Article  Google Scholar 

  • Bayindir M, Bolger F, Say B (2017) An investigation of the role of some person and situation variables in multiple cue probability learning. Q J Exp Psychol 70(1):36–52

    Article  Google Scholar 

  • Bazerman MH, Moore DA (2008) Judgment in managerial decision making, 7th edn. Wiley, New York

    Google Scholar 

  • Bolger F, Rowe G (2015a) The aggregation of expert judgment: do good things come to those who weight? Risk Anal 35(1):5–11

    Article  Google Scholar 

  • Bolger F, Rowe G (2015b) There is data, and then there is data: only experimental evidence will determine the utility of differential weighting of expert judgment. Risk Anal 35(1):21–26

    Article  Google Scholar 

  • Bolger F, Wright G (1993) Coherence and calibration in expert probability judgement. OMEGA Int J Manag Sci 21:629–644

    Article  Google Scholar 

  • Bolger F, Wright G (1994) Assessing the quality of expert judgment: issues and analysis. Decis Support Syst 11:1–24

    Article  Google Scholar 

  • Bolger F, Wright G (2011) Improving the Delphi process: lessons from social psychological research. Technol Forecast Soc Chang 78:1500–1513

    Article  Google Scholar 

  • Bolger F, Wright G (2017) Use of expert knowledge to anticipate the future: issues, analysis and directions. Int J Forecast 33(1):230–243

    Article  Google Scholar 

  • Bolger F, Wright G, Rowe G, Gammack J, Wood R (1989) LUST for life: developing expert systems for life assurance underwriting. In: Shadbolt N (ed) Research and development in expert systems VI. Cambridge University Press, Cambridge, pp 128–139

    Google Scholar 

  • Bolger F, Hanea A, Mosbach-Schulz O, Oakley J, O’Hagan A, Rowe G, Wenholt M (2014) Guidance on expert knowledge elicitation in food and feed safety risk assessment. European Food Safety Authority (EFSA), Parma

    Google Scholar 

  • Brenner LA (2003) A random support model of the calibration of subjective probabilities. Organ Behav Hum Decis Process 90(1):87–110

    Article  Google Scholar 

  • Brunswik E (1955) Representative design and probabilistic theory in a functional psychology. Psychol Rev 62(3):193

    Article  Google Scholar 

  • Budescu DV, Chen E (2015) Identifying expertise to extract the wisdom of crowds. Manag Sci 61(2):267–280

    Article  Google Scholar 

  • Burgman MA, McBride M, Ashton R, Speirs-Bridge A, Flander L et al (2011) Expert status and performance. PLoS One 6(7):E22998

    Article  Google Scholar 

  • Chase WG, Simon HA (1973) Perception in chess. Cogn Psychol 4(1):55–81

    Article  Google Scholar 

  • Clemen RT (2008) Comment on Cooke’s classical method. Reliab Eng Syst Saf 93:760–765

    Article  Google Scholar 

  • Cooke RM (2014) Validating expert judgments with the classical model. In: Martini C, Boumans M (eds) Experts and consensus in social science—critical perspectives from economics, sociology, politics, and philosophy. Ethical economy—studies in economic ethics and philosophy. Springer, Heidelberg, pp 191–121

    Google Scholar 

  • Cooke RM, Goossens LLHJ (2008) TU Delft expert judgment database. Reliab Eng Syst Saf 93:657–674

    Article  Google Scholar 

  • Cooke RM, ElSaadany S, Huang X (2008) On the performance of social network and likelihood-based expert weighting schemes. Reliab Eng Syst Saf 93(5):745–756

    Google Scholar 

  • Dalgleish LI (1988) Decision making in child abuse cases: Applications of social judgment theory and signal detection theory. In: Brehmer B, Joyce CB (eds) Human judgment: the SJT view. Elsevier, Amsterdam, pp 317–360

    Chapter  Google Scholar 

  • Dalkey NC (1975) Toward a theory of group estimation. In: Linstone HA, Turoff M (eds) The Delphi method: techniques and applications. Addison-Wesley, Reading, MA, pp 236–261

    Google Scholar 

  • De Bondt WF, Thaler RH (1989) Anomalies: a mean-reverting walk down Wall Street. J Econ Perspect 3(1):189–202

    Article  Google Scholar 

  • De Groot AD (1965) Thought and choice in chess. The Hague, Mouton

    Google Scholar 

  • Dhami MK, Wallsten TS (2005) Interpersonal comparison of subjective probabilities: toward translating linguistic probabilities. Mem Cogn 33(6):1057–1068

    Article  Google Scholar 

  • Dror IE (2011) The paradox of human expertise: why experts get it wrong. In: Kapur N (ed) The paradoxical brain. Cambridge University Press, Cambridge, pp 177–188

    Chapter  Google Scholar 

  • Eden C (1988) Cognitive mapping. Eur J Oper Res 36(1):1–13

    Article  Google Scholar 

  • Edwards W (1968) Conservatism in human information processing. In: Kleinmuntz B (ed) Formal representation of human judgment, CMU cognition series, vol 3. Wiley, New York, pp 17–51

    Google Scholar 

  • Eggstaff JW, Mazzuchi TA, Sarkani S (2014) The effect of the number of seed variables on the performance of Cooke’s classical model. Reliab Eng Syst Saf 121:72–82

    Article  Google Scholar 

  • Ellis HC, Ashbrook PW (1989) The state of mood and memory research: a selective review. J Soc Behav Pers 4(2):1–21

    Google Scholar 

  • European Food Safety Authority (2013) Technical meeting of the EFSA Scientific Network on EFSA Scientific. Network for risk assessment in Animal Health and Welfare – Risk of introduction of Rift Valley fever into the Southern Mediterranean area through undocumented movement of infected animals. EFSA Supporting Documents, 10 (4)

    Google Scholar 

  • Fama EF (1965) The behaviour of stock market prices. J Bus 38:34–105

    Article  Google Scholar 

  • Ferrell WR, McGoey PJ (1980) A model of calibration for subjective probabilities. Organ Behav Hum Perform 26(1):32–53

    Article  Google Scholar 

  • Frederick S (2005) Cognitive reflection and decision making. J Econ Perspect 19(4):25–42

    Article  Google Scholar 

  • Genre V, Kenny G, Meyler A, Timmermann A (2013) Combining expert forecasts: can anything beat the simple average? Int J Forecast 29:108–121

    Article  Google Scholar 

  • Germain ML (2006) Development and preliminary validation of a psychometric measure of expertise: the generalized expertise measure (GEM). Unpublished doctoral dissertation. Barry University, Florida, USA

    Google Scholar 

  • Germain M-L, Tejeda MJ (2012) A preliminary exploration on the measurement of expertise: an initial development of a psychometric scale. Hum Resour Dev Q 23(2):203–232

    Article  Google Scholar 

  • Gibbons AM, Sniezek JA, Dalal RS (2003) Antecedents and consequences of unsolicited versus explicitly solicited advice. In: Budescu D (Chair), Symposium in Honor of Janet Sniezek. Symposium presented at the annual meeting of the Society for Judgment and Decision Making, Vancouver, BC (2003 November)

    Google Scholar 

  • Gigerenzer G, Hoffrage U (1995) How to improve Bayesian reasoning without instruction: frequency formats. Psychol Rev 102(4):684

    Article  Google Scholar 

  • Gigerenzer G, Hoffrage U, Kleinbölting H (1991) Probabilistic mental models: a Brunswikian theory of confidence. Psychol Rev 98(4):506–528

    Article  Google Scholar 

  • Goldberg LR (1970) Man versus model of man: a rationale, plus some evidence, for a method of improving on clinical inferences. Psychol Bull 73(6):422

    Article  Google Scholar 

  • Griffin D, Brenner L (2004) Perspectives on probability judgment calibration. In: Koehler DJ, Harvey N (eds) Blackwell handbook of judgment and decision making. Blackwell, Malden (MA), pp 177–199

    Chapter  Google Scholar 

  • Griffin D, Tversky A (1992) The weighing of evidence and the determinants of confidence. Cogn Psychol 24(3):411–435

    Article  Google Scholar 

  • Hardman DK (2009) Judgment and decision making: psychological perspectives. Wiley, Chichester

    Google Scholar 

  • Hayes-Roth F, Waterman D, Lenat D (1983) Building expert systems. Addison-Wesley, Boston

    Google Scholar 

  • Hertwig R, Barron G, Weber EU, Erev I (2004) Decisions from experience and the effect of rare events in risky choice. Psychol Sci 15(8):534–539

    Article  Google Scholar 

  • Hodge R, Evans M, Marshall J, Quigley J, Walls L (2001) Eliciting engineering knowledge about reliability during design-lessons learnt from implementation. Qual Reliab Eng Int 17(3):169–179

    Article  Google Scholar 

  • Hoefling A, Strack F (2010) Hunger induced changes in food choice. When beggars cannot be choosers even if they are allowed to choose. Appetite 54(3):603–606

    Article  Google Scholar 

  • Howard RA, Matheson JE (2005) Influence diagrams. Decis Anal 3:127–143

    Article  Google Scholar 

  • Isen AM, Erez A (2002) The influence of positive affect on the components of expectancy motivation. J Appl Psychol 87(6):1055–1067

    Article  Google Scholar 

  • Ivlev I, Kneppo P, Bartak M (2015) Method for selecting expert groups and determining the importance of experts’ judgments for the purpose of managerial decision-making tasks in health systems. Ekon Manag 2(18):57–72

    Google Scholar 

  • Janis IL (1982) Groupthink: psychological studies of policy decisions and fiascoes. Houghton Mifflin, Boston

    Google Scholar 

  • Juslin P (1993) An explanation of the hard-easy effect in studies of realism of confidence in one’s general knowledge. Eur J Cogn Psychol 5(1):55–71

    Article  Google Scholar 

  • Juslin P (1994) The overconfidence phenomenon as a consequence of informal experimenter-guided selection of almanac items. Organ Behav Hum Decis Process 57(2):226–246

    Article  Google Scholar 

  • Keren G (1991) Calibration and probability judgements: conceptual and methodological issues. Acta Psychol 77(3):217–273

    Article  Google Scholar 

  • Klahr D, Langley P, Neches R (1987) Production system models of learning and development. MIT Press, Cambridge, MA

    Google Scholar 

  • Klein G (1998) Sources of power: how people make decisions. MIT Press, Cambridge, MA

    Google Scholar 

  • Kleindorfer PR, Kunreuther H, Schoemaker PJ (1993) Decision sciences: an integrative perspective. Cambridge University Press, Cambridge, p 484

    Book  Google Scholar 

  • Kliger D, Levy O (2003) Mood and judgment of subjective probabilities: evidence from the US index option market. Eur Finan Rev 7(2):235–248

    Article  Google Scholar 

  • Koehler D, Brenner L, Griffin D (2002) The calibration of expert judgment: heuristics and biases beyond the laboratory. In: Gilovich T, Griffin D, Kahneman D (eds) Heuristics and biases: the psychology of intuitive judgment. Cambridge University Press, Cambridge, pp 686–715

    Chapter  Google Scholar 

  • Koriat A, Lichtenstein S, Fischhoff B (1980) Reasons for confidence. J Exp Psychol Hum Learn Mem 6(2):107–118

    Article  Google Scholar 

  • Lave J, Wenger E (1991) Situated learning: legitimate peripheral participation. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Lichtenstein S, Fischhoff B, Phillips L (1982) Calibration of probabilities: the state of the art to 1980. In: Kahneman D, Slovic P, Tversky A (eds) Judgment under uncertainty: heuristics and biases. Cambridge University Press, Cambridge, pp 306–334

    Chapter  Google Scholar 

  • Lin SW, Bier VM (2008) A study of expert overconfidence. Reliab Eng Syst Saf 93:711–721

    Article  Google Scholar 

  • Lin SW, Cheng CH (2009) The reliability of aggregated probability judgments obtained through Cooke’s classical method. J Model Manag 4:149–161

    Article  Google Scholar 

  • Lohr S (2009) Sampling: design and analysis. Nelson Education, Toronto

    Google Scholar 

  • MacGregor DG, Lichtenstein S (1991) Problem structuring aids for quantitative estimation. J Behav Decis Mak 4:101–116

    Article  Google Scholar 

  • MacGregor DG, Lichtenstein S, Slovic P (1988) Structuring knowledge retrieval: an analysis of decomposed quantitative judgments. Organ Behav Hum Decis Process 42:303–323

    Article  Google Scholar 

  • MacIntyre A (2007) After virtue: a study in moral theory, 3rd edn. Duckworth, London

    Google Scholar 

  • Malkiel BG (2011, October) The efficient-market hypothesis and the financial crisis. In: Rethinking finance: perspectives on the crisis (Proceedings of a conference). Russel Sage Foundation

    Google Scholar 

  • Mannes AE, Soll JB, Larrick RP (2014) The wisdom of select crowds. J Pers Soc Psychol 107(2):76–299

    Article  Google Scholar 

  • Meyer MA, Booker JM (1991) Eliciting and analyzing expert judgment: a practical guide. Academic, London

    Google Scholar 

  • Moore G, Beadle R (2006) In search of organizational virtue in business: agents, goods, practices, institutions and environments. Organ Stud 27:369–389

    Article  Google Scholar 

  • Moreira C, Wichert A (2013) Finding academic experts on a multisensor approach using Shannon’s entropy. Expert Syst Appl 40:5740–5754

    Article  Google Scholar 

  • Murphy AH, Brown BG (1985) A comparative evaluation of objective and subjective weather forecasts in the United States. In: Wright G (ed) Behavioral decision making. Plenum, New York, pp 329–359

    Chapter  Google Scholar 

  • Newell A, Simon HA (1972) Human problem solving. Prentice-Hall, Englewood Cliffs, NJ

    Google Scholar 

  • Newell A, Shaw JC, Simon HA (1959) The processes of creative thinking. Rand Corporation, Santa Monica, CA

    Google Scholar 

  • Nisbett RE, Wilson TD (1977) Telling more than we can know: verbal reports on mental processes. Psychol Rev 84(3):231–259

    Article  Google Scholar 

  • Oliver RM, Smith JQ (1990) Influence diagrams, belief nets and decision analysis. Wiley, New York

    Google Scholar 

  • Olsson H (2014) Measuring overconfidence: methodological problems and statistical artifacts. J Bus Res 67:1766–1770

    Article  Google Scholar 

  • Oskamp S (1965) Overconfidence in case study judgments. J Consult Psychol 29:261–265

    Article  Google Scholar 

  • Parenté FJ, Anderson-Parenté JK (1987) Delphi inquiry systems. In: Wright G, Ayton P (eds) Judgmental forecasting. Wiley, Chichester, pp 129–156

    Google Scholar 

  • Phillips LD (1987) On the adequacy of judgmental forecasts. In: Wright G, Ayton P (eds) Judgmental forecasting. Wiley, Oxford, pp 11–30

    Google Scholar 

  • Phillips JM (1999) Antecedents of leader utilization of staff input in decision-making teams. Organ Behav Hum Decis Process 77:215–242

    Article  Google Scholar 

  • Price PC, Stone ER (2004) Intuitive evaluation of likelihood judgment producers: evidence for a confidence heuristic. J Behav Decis Mak 17(1):39–57

    Article  Google Scholar 

  • Rakow T, Newell BR (2010) Degrees of uncertainty: an overview and framework for future research on experience-based choice. J Behav Decis Mak 23(1):1–14

    Article  Google Scholar 

  • Rowe G, Bolger F (2016) The identification of food safety priorities using the Delphi technique. EFSA Supporting Publications, 13 (3). EN-1007, 141 pp

    Google Scholar 

  • Rowe G, Wright G (1996) The impact of task characteristics on the performance of structured group forecasting techniques. Int J Forecast 12:73–90

    Article  Google Scholar 

  • Rowe G, Wright G (2001) Differences in expert and lay judgments of risk: myth or reality? Risk Anal 21(2):341–356

    Article  Google Scholar 

  • Rowe G, Wright G, McColl A (2005) Judgment changes during Delphi-like procedures: the role of majority influence, expertise, and confidence. Technol Forecast Soc Chang 72:217–238

    Article  Google Scholar 

  • Saunders EMJ (1993) Stock prices and Wall Street weather. Am Econ Rev 83:1337–1345

    Google Scholar 

  • Solomon I, Ariyo A, Tomasini LA (1985) Contextual effects on the calibration of probabilistic judgments. J Appl Psychol 70:528–532

    Article  Google Scholar 

  • Tetlock P, Gardner D (2016) Superforecasting: the art and science of prediction. Random House, New York

    Google Scholar 

  • Toulmin S (1958) The uses of argument. Cambridge University Press, Cambridge, p 272

    Google Scholar 

  • Tversky A, Koehler DJ (1994) Support theory: a nonextensional representation of subjective probability. Psychol Rev 101:547–567

    Article  Google Scholar 

  • Walls L, Quigley J, Marshall J (2006) Modeling to support reliability enhancement during product development with applications in the UK aerospace industry. IEEE Trans Eng Manag 53(2):263–274

    Article  Google Scholar 

  • Wallsten TS, Budescu DV (1995) A review of human linguistic probability processing: general principles and empirical evidence. Knowl Eng Rev 10(1):43–62

    Article  Google Scholar 

  • Waterman DA, Hayes-Roth F (1978) Pattern-directed inference systems. Academic, New York

    Google Scholar 

  • Wenger E (1998) Communities of practice: learning, meaning and identity. Cambridge University Press, Cambridge

    Book  Google Scholar 

  • Wright G, Rowe G, Bolger F, Gammack J (1994) Coherence, calibration and expertise in judgmental probability forecasting. Organ Behav Hum Decis Process 57:1–25

    Article  Google Scholar 

  • Wright G, Bolger F, Rowe G (2002) An empirical test of the relative validity of expert and lay judgments of risk. Risk Anal 22:1107–1122

    Article  Google Scholar 

  • Wright G, Rowe G, McColl A (2004) A framework for future study of expert and lay differences in the judgment of risk. Risk Decis Policy 9(2):91–106

    Article  Google Scholar 

Download references

Acknowledgements

I wish to acknowledge the support of the European Food Safety Authority (EFSA) who convened the “Working Group on Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment”: the research conducted for which laid the foundations for this chapter. I also wish to thank Working Group members, Anca Hanea, Anthony O‘Hagan, Jeremy Oakley, Gene Rowe and Meike Wentholt; and EFSA staff, Elisa Aiassa, Fulvio Barizzone, Eugen Christoph, Andrea Gervelmeyer, Olaf Mosbach-Schulz, and Sara Tramontini for their contributions to the original research. Finally, I am grateful to David Budescu, John Quigley, and Gene Rowe for their comments on earlier drafts of this manuscript.

Author information

Authors and Affiliations

  1. Strathclyde Business School, Strategy & Organisation, 199 Cathedral Street, G4 0QU, Glasgow, UK

    Fergus Bolger

Authors
  1. Fergus Bolger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fergus Bolger .

Editor information

Editors and Affiliations

  1. Faculty of Economics, CeBER and INESC Coimbra, University of Coimbra, Coimbra, Portugal

    Luis C. Dias

  2. Department of Management Science, University of Strathclyde, Scotland, United Kingdom

    Alec Morton

  3. Department of Management Science, University of Strathclyde, Scotland, United Kingdom

    John Quigley

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Bolger, F. (2018). The Selection of Experts for (Probabilistic) Expert Knowledge Elicitation. In: Dias, L., Morton, A., Quigley, J. (eds) Elicitation. International Series in Operations Research & Management Science, vol 261. Springer, Cham. https://doi.org/10.1007/978-3-319-65052-4_16

Download citation

Publish with us

Policies and ethics

Search

Navigation