Advertisement

Aggregating expert judgement

  • Simon FrenchEmail author
Survey

Abstract

In a paper written some 25 years ago, I distinguished three contexts in which one might wish to combine expert judgements of uncertainty: the expert problem, the group decision problem and the textbook problem. Over the intervening years much has been written on the first two, which have the focus of a single decision context, but little on the third, though the closely related field of meta-analysis has developed considerably. With many developments in internet technology, particularly in relation to interactivity and communication, the textbook problem is gaining in importance since data and expert judgements can be made available over the web to be used by many different individuals to shape their own beliefs in many different contexts. Moreover, applications such as web-based decision support, e-participation and e-democracy are making algorithmic ‘solutions’ to the group decision problem attractive, despite many results showing we know that such solutions are, at best, rare and, at worst, illusory. In this paper I survey developments since my earlier paper and note some unresolved issues. Then I turn to how expert judgement might be used within web-based group decision support, as well as in e-participation and e-democracy contexts. The latter points to a growing importance of the textbook problem and suggests that Cooke’s principles for scientific reporting of expert judgement studies may need enhancing for such studies to be used by a wider audience.

Keywords

Aggregation of expert judgement Cooke’s Principles e-Democracy e-Participation Expert judgement Expert problem Group decision problem Meta-analysis Textbook problem Web-based group decision support systems (wGDSS) 

Mathematics Subject Classification (2000)

62C10 91B06 91B10 91B12 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Adhikari, S., Bayley, C., Bedford, T., Busby, J.S., Cliffe, A., Devgun, G., Eid, M., French, S., Keshvala, R., Pollard, S., Soane, E., Tracy, D., Wu, S.: Human Reliability Analysis: A Review and Critique. Manchester Business School, Manchester (2008)Google Scholar
  2. 2.
    Armstrong, J.S.: Combining forecasts. In: Armstrong, J.S. (ed.) Principles of Forecasting: a Handbook for Researchers and Practitioners, pp. 417–439. Kluwer, Norwell (2001)Google Scholar
  3. 3.
    Arnell N.W., Tompkins E.L., Adger W.N.: Eliciting information from experts on the likelihood of rapid climate change. Risk Anal. 25(6), 1419–1431 (2005)CrossRefGoogle Scholar
  4. 4.
    Arrow K.J.: Social Choice and Individual Values. Wiley, New York (1963)Google Scholar
  5. 5.
    Ashby D., Smith A.F.M.: Evidence-based medicine as Bayesian decision-making. Stat. Med. 19, 3291–3305 (2000)CrossRefGoogle Scholar
  6. 6.
    Aspinal W.: A route to more tractiable expert advice. Nature 463, 294–295 (2010)CrossRefGoogle Scholar
  7. 7.
    Bacharach M.: Group decisions in the face of differences of opinion. Manag. Sci. 22(2), 182–191 (1975)MathSciNetzbMATHCrossRefGoogle Scholar
  8. 8.
    Bayley, C., French, S.: Public participation: comparing approaches. J. Risk Res. (2010, in press)Google Scholar
  9. 9.
    Bedford T., Cooke R.: Probabilistic Risk Analysis: Foundations and Methods. Cambridge University Press, Cambridge (2001)zbMATHGoogle Scholar
  10. 10.
    Bedford T., Quigley J., Walls L.: Expert elicitation for reliable system design. Stat. Sci. 21(4), 428–450 (2006)MathSciNetzbMATHCrossRefGoogle Scholar
  11. 11.
    Beierle, T., Cayford, J.: Democracy in Practice: Public Participation in Environmental Decisions. Resources for the Future (2002)Google Scholar
  12. 12.
    Bennett, P.G., Calman, K.C. (eds): Risk Communication and Public Health: Policy Science and Participation. Oxford University Press, Oxford (1999)Google Scholar
  13. 13.
    Bennett, P.G., Calman, K.C., Curtis, S., Fischbacher-Smith, D. (eds): Risk Communication and Public Health, 2nd edn. Oxford University Press, Oxford (2010)Google Scholar
  14. 14.
    Bier V.M.: Implications of the research on expert overconfidence and dependence. Reliab. Eng. Syst. Saf. 85, 321–329 (2004)CrossRefGoogle Scholar
  15. 15.
    Bolger F., Wright G.: Assessing the quality of expert judgment: issues and analysis. Decis. Support Syst. 11(1), 1–24 (1994)CrossRefGoogle Scholar
  16. 16.
    Booker J.M., Meyer M.A.: Sources and effects of interexpert correlation: an empirical study. IEEE Trans. Syst. Man Cybern. 18, 135–142 (1988)CrossRefGoogle Scholar
  17. 17.
    Boone I., Van der Stede Y., Bollaerts K., Messens W., Vose D., Daube G., Aerts M., Mintiens K.: Expert judgement in a risk assessment model for Salmonella spp. in pork: the performance of different weighting schemes. Prev. Vet. Med. 92(3), 224–234 (2009)CrossRefGoogle Scholar
  18. 18.
    Bordley R.F.: Combining the opinions of experts who partition events differently. Decis. Anal. 6(1), 38–46 (2009)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Borenstein M., Hedges L.V., Higgins J.P.T., Rothstein H.R.: Introduction to Meta-Analysis. Wiley, Chichester (2009)zbMATHCrossRefGoogle Scholar
  20. 20.
    Burns W.J., Clemen R.T.: Covariance structure models and influence diagrams. Manag. Sci. 39(7), 816–834 (1993)CrossRefGoogle Scholar
  21. 21.
    Clemen R.T.: Calibration and the aggregation of probabilities. Manag. Sci. 32(3), 312–314 (1986)CrossRefGoogle Scholar
  22. 22.
    Clemen R.T.: Combining forecasts. Int. J. Forecast. 5, 559–583 (1989)CrossRefGoogle Scholar
  23. 23.
    Clemen R.T., Winkler R.L.: Aggregating point estimates: a flexible modelling approach. Manag. Sci. 39, 501–515 (1993)zbMATHCrossRefGoogle Scholar
  24. 24.
    Clemen R.T., Reilly T.: Correlations and copulas for decision and risk analyses. Manag. Sci. 45(2), 208–224 (1999)CrossRefGoogle Scholar
  25. 25.
    Clemen, R.T., Winkler, R.L.: Combining probability distributions from experts in risk analysis. Risk Anal. 19 (1999)Google Scholar
  26. 26.
    Clemen R.T., Fischer G.W., Winkler R.L.: Assessing dependence: some experimental results. Manag. Sci. 46(8), 1100–1115 (2000)CrossRefGoogle Scholar
  27. 27.
    Clemen, R.T., Lichtendahl, K.C.: Debiasing expert overconfidence: a Bayesian calibration model. PSAM6, San Juan, Puerto Rico (2002)Google Scholar
  28. 28.
    Clemen, R.T., Winkler, R.L.: Aggregating probability distributions. In: Edwards, W., Miles, R.F., Von Winterfeldt, D. (eds.) Advances in Decision Analysis: From Foundations to Applications, pp. 154–176. Cambridge University Press, Cambridge (2007)Google Scholar
  29. 29.
    Clemen R.T.: Comment on Cooke’s classical method. Reliab. Eng. Syst. Saf. 93(5), 760–765 (2008)Google Scholar
  30. 30.
    Cleveland W.S.: The Elements of Graphing Data. Murray Hill, New Jersey (1994)Google Scholar
  31. 31.
    Conitzer, V., Rothe, J. (eds.) COMSOC2010: Proceedings of the Third International Workshop on Computational Social Choice, September 13–16, 2010. Dusseldorf University Press, Dusseldorf (2010)Google Scholar
  32. 32.
    Cooke R.M.: Experts in Uncertainty. Oxford University Press, Oxford (1991)Google Scholar
  33. 33.
    Cooke, R.M., Bedford, T., Kraan, B.: Report on the Benchmark Workshop on Uncertainty/Sensitivity Analysis Codes. European Safety and Reliability Association. Copies from R. Cooke, Mathematics and Informatics. TU Delft, Delft (1997)Google Scholar
  34. 34.
    Cooke R.M., Goossens L.H.J.: Expert judgement elicitation for risk assessments of critical infrastructures. J. Risk Res. 7(6), 643–656 (2004)CrossRefGoogle Scholar
  35. 35.
    Cooke, R.M., Probst, K.N.: Highlights of the Expert Judgement Policy Symposium and Technical Workshop. Resources for the Future, Washington, DC (2006)Google Scholar
  36. 36.
    Cooke R.M.: Expert judgement studies. Reliab. Eng. Syst. Saf. 93, 655–777 (2007)CrossRefGoogle Scholar
  37. 37.
    Cooke R.M., ElSaadany S., Xinzheng H.: On the performance of social network and likelihood based expert weighting schemes. Reliab. Eng. Syst. Saf. 93(5), 745–756 (2007)CrossRefGoogle Scholar
  38. 38.
    Cooke R.M., Goossens L.H.J.: TU delft expert judgement database. Reliab. Eng. Syst. Saf. 93(5), 657–674 (2007)CrossRefGoogle Scholar
  39. 39.
    Cooke R.M.: Response to the reliability of aggregated probability judgments obtained through Cooke’s classical model. TU Delft, Delft (2008)Google Scholar
  40. 40.
    Dalkey N., Helmer O.: An experimental application of the Delphi method to the use of experts. Manag. Sci. 9(3), 458–467 (1963)CrossRefGoogle Scholar
  41. 41.
    Dawid A.P., DeGroot M.H., Mortera J.: Coherent combination of experts’ opinions (with discussion). Test 4(2), 263–313 (1995)MathSciNetzbMATHCrossRefGoogle Scholar
  42. 42.
    De Finetti B.: Theory of Probability. Wiley, Chichester (1974)zbMATHGoogle Scholar
  43. 43.
    DeGroot M.H., Bayarri M.J.: What Bayesians expect of each other. J. Am. Stat. Assoc. 86(416), 924–932 (1991)MathSciNetzbMATHCrossRefGoogle Scholar
  44. 44.
    Department of Health: Communicating About Risks to Public Health: Pointers to Good Practice. HMSO, London (1998)Google Scholar
  45. 45.
    DeWispelare A.R., Herren L.T., Clemen R.T.: The use of probability elicitation in the high-level nuclear waste regulation program. Int. J. Forecast. 11, 5–24 (1995)CrossRefGoogle Scholar
  46. 46.
    Dryzek J.S., List C.: Social choice theory and deliberative democracy: a reconciliation. Br. J. Political Sci. 33(1), 1–28 (2003)CrossRefGoogle Scholar
  47. 47.
    Eden, C., Radford, J. (eds): Tackling Strategic Problems: the Role of Group Decision Support. Sage, London (1990)Google Scholar
  48. 48.
    Ehrhardt J., Brown J., French S., Kelly G.N., Mikkelsen T., Muller H: RODOS: decision making support for off-site emergency management after nuclear accidents. Kerntechnik 62, 122–128 (1997)Google Scholar
  49. 49.
    Ehrhardt J., Weiss A.: RODOS: Decision Support for Off-Site Nuclear Emergency Management in Europe. EUR19144EN. European Community, Luxembourg (2000)Google Scholar
  50. 50.
    Faria A., Smith J.Q.: Conditionally externally Bayesian pooling operators in chain graphs. Ann. Stat. 25(4), 1740–1761 (1997)MathSciNetzbMATHCrossRefGoogle Scholar
  51. 51.
    Franco A., Shaw D., Westcombe M.: Problem structuring methods I. J. Oper. Res. Soc. 57, 757–878 (2006)CrossRefGoogle Scholar
  52. 52.
    Franco A., Shaw D., Westcombe M.: Problem structuring methods II. J. Oper. Res. Soc. 58, 545–682 (2007)CrossRefGoogle Scholar
  53. 53.
    Franke G.R.: Applications of meta-analysis for marketing and public policy: a review. J. Public Policy Market. 20(2), 186–200 (2001)CrossRefGoogle Scholar
  54. 54.
    French S.: Updating of belief in the light of someone else’s opinion. J. R. Stat. Soc. A 143, 43–48 (1980)MathSciNetzbMATHCrossRefGoogle Scholar
  55. 55.
    French S.: Consensus of opinion. Eur. J. Oper. Res. 7, 332–340 (1981)MathSciNetzbMATHCrossRefGoogle Scholar
  56. 56.
    French, S.: Group consensus probability distributions: a critical survey. In: Bernardo, J.M., DeGroot, M.H., Lindley, D.V., Smith, A.F.M. (eds.) Bayesian Statistics, vol. 2, pp. 183–201. North-Holland, Amsterdam (1985)Google Scholar
  57. 57.
    French S.: Calibration and the expert problem. Manag. Sci. 32, 315–321 (1986)CrossRefGoogle Scholar
  58. 58.
    French, S.: Conflict of belief: when advisers disagree. In: Bennett, P.G. (ed.) Analysing Conflict and its Resolution: Some Mathematical Contributions, pp. 93–111. Oxford University Press, Oxford (1987)Google Scholar
  59. 59.
    French, S., Cooke, R.M., Vogt, F.: The use of expert judgement in the context of a physical model. In: Bernardo, J.M., Berger, J.O., Dawid, A.P., Smith, A.F.M. (eds.) Bayesian Statistics, vol. IV, pp. 617–624. Oxford University Press, Oxford (1991)Google Scholar
  60. 60.
    French S., Rios Insua D.: Statistical Decision Theory. Arnold, London (2000)Google Scholar
  61. 61.
    French S., Maule A.J., Mythen G., Wales C.: Trust and Risk Communications. Manchester Business School, Manchester (2002)Google Scholar
  62. 62.
    French S.: Modelling, making inferences and making decisions: the roles of sensitivity analysis. TOP 11(2), 229–252 (2003)MathSciNetzbMATHCrossRefGoogle Scholar
  63. 63.
    French, S., Battson, A.: Review and implications of research on the cognitive understanding of uncertainties relating to spatial and temporal plots. ENSEMBLE Project Report, Manchester Business School, Manchester (2003)Google Scholar
  64. 64.
    French, S., Bayley, C.: An analysis of the ENSEMBLE exercises from the perspective of decision making. ENSEMBLE Project Report, Manchester Business School, Manchester (2003)Google Scholar
  65. 65.
    French, S., Rios Insua, D., Ruggeri, F.: e-Participation and decision analysis. Decis. Anal. (2006, under revision)Google Scholar
  66. 66.
    French S.: Web-enabled strategic GDSS, e-democracy and Arrow’s Theorem: a Bayesian perspective. Decis. Support Syst. 43, 1476–1484 (2007)CrossRefGoogle Scholar
  67. 67.
    French S., Carter E., Niculae C.: Decision support in nuclear and radiological emergency situations: are we too focused on models and technology?. Int. J. Emerg. Manag. 4(3), 421–441 (2007)CrossRefGoogle Scholar
  68. 68.
    French S., Rios Insua D., Ruggeri F.: e-participation and decision analysis. Decis. Anal. 4(4), 1–16 (2007)CrossRefGoogle Scholar
  69. 69.
    French S., Maule A.J., Papamichail K.N.: Decision Behaviour, Analysis and Support. Cambridge University Press, Cambridge (2009)CrossRefGoogle Scholar
  70. 70.
    French, S., Bedford, T., Pollard, S., Soane, E.: Human Reliability Analysis: a critique and review for the management of risk. Saf. Sci. (2010, in press)Google Scholar
  71. 71.
    Galmarini S., Bianconi R., Klug W., Mikkelsen T., Addis R., Andronopoulos S., Astrup P., Baklanov A., Bartniki J., Bartzis J., Bellasio R., Bompay F., Buckley R., Bouzom M., Champion H., D’Amours R., Davakis E., Eleveld H., Geertsema G.T., Glaab H., Kollax M., Ilvonen M., Manning A., Pechinger U., Persson C., Polreich E., Ptempski S., Prodanova M., Saltbones J., Slaper H., Sofiev M.A., Syrakov D., Sorenson J.H., Vander Auwera L., Valkama I., Zelazny R.: Can the confidence in long range atmospheric transport models be increased? The pan-European experience of ENSEMBLE. Radiat. Prot. Dosim. 109(1–2), 19–24 (2004)CrossRefGoogle Scholar
  72. 72.
    Garthwaite P.H., Kadane J.B., O’Hagan A.: Statistical methods for eliciting probability distributions. J. Am. Stat. Assoc. 100(470), 680–701 (2005)MathSciNetzbMATHCrossRefGoogle Scholar
  73. 73.
    Genest C., Zidek J.V.: Combining probability distributions: a critique and annotated bibliography. Stat. Sci. 1, 114–148 (1986)MathSciNetCrossRefGoogle Scholar
  74. 74.
    Genest C., Wagner C.G.: Further evidence against independence preservation in expert judgement synthesis. Aequationes Mathematicae 32, 74–86 (1987)MathSciNetzbMATHCrossRefGoogle Scholar
  75. 75.
    Geyskens, I., Krishnan, R., Steenkamp, J.-B.E.M., Cunha, P.V.: A review and evaluation of meta-analysis practices in management research. J. Manag. (2009). doi: 10.1177/0149206308328501
  76. 76.
    Gigerenzer G.: Reckoning with Risk: Learning to live with Uncertainty. Penguin Books, Harmondsworth (2002)Google Scholar
  77. 77.
    Gilovich, T., Griffin, D., Kahneman, D. (eds): Heuristics and Biases: The Psychology of Intuitive Judgment. Cambridge University Press, Cambridge (2002)Google Scholar
  78. 78.
    Givens G.H., Tweedie Smith, Tweedie Smith: Publication bias in meta-analysis: a Bayesian data-augmentation approach to account for issues exemplified in the passive smoking debate. Stat. Sci. 12(4), 221–250 (1997)CrossRefGoogle Scholar
  79. 79.
    Goldstein M., Rougier J.C.: Reified Bayesian Modelling and inference for physical systems (with discussion). J. Stat. Plan. Inference 139, 1221–1256 (2009)MathSciNetzbMATHCrossRefGoogle Scholar
  80. 80.
    Goldstein, M.: External Bayesian analysis for computer simulators. In: Bernardo, J.M., Bayarri, M.J., Bergeret, J.O., et al. (eds.) Bayesian Statistics, vol. 9. Oxford University Press, Oxford (2010, in press)Google Scholar
  81. 81.
    Goossens L.H.J., Kelly G.N.: Special issue: expert judgement and accident consequence uncertainty analysis. Radiat. Prot. Dosim. 90(3), 293–381 (2000)Google Scholar
  82. 82.
    Gregory R.S., Fischhoff B., McDaniels T.: Acceptable input: using decision analysis to guide public policy deliberations. Decis. Anal. 2(1), 4–16 (2005)CrossRefGoogle Scholar
  83. 83.
    Hartung J., Knapp J., Sinha B.K.: Statistical Meta-Analysis with Applications. Wiley, Hoboken (2008)CrossRefGoogle Scholar
  84. 84.
    Hastie, R.: Experimental evidence of group accuracy. In: Grofman, B., Owen, G. (eds.) Information Pooling and Group Decision Making, pp. 129–157. JAI Press, Greenwich (1986)Google Scholar
  85. 85.
    Hockey G.R.J., Maule A.J., Clough P.J., Bdzola L.: Effects of negative mood on risk in everyday decision making. Cogn. Emot. 14, 823–856 (2000)CrossRefGoogle Scholar
  86. 86.
    Hodge J.K., Klima R.E.: The Mathematics of Voting and Elections: a Hands-On Approach. American Mathematical Society, Rhode Island (2005)zbMATHGoogle Scholar
  87. 87.
    Hora, S.: Eliciting probabilities from experts. In: Edwards, W., Miles, R.F., Von Winterfeldt, D. (eds.) Advances in Decision Analysis: From Foundations to Applications, pp. 129–153. Cambridge University Press, Cambridge (2007)Google Scholar
  88. 88.
    Hora S.C.: Probability judgements for continuous quantities: linear combinations and calibration. Manag. Sci. 50(5), 597–604 (2004)CrossRefGoogle Scholar
  89. 89.
    Hora, S.C.: Expert judgement. In: Melnick, E.L., Everitt, B.S.: Encylcopedia of Quantitative Risk Analysis and Assessment, pp. 667–676. Wiley, Chichester (2008)Google Scholar
  90. 90.
    Jacobs R.A.: Methods for combining experts’ probability assessments. Neural Comput. 7, 867–888 (1995)CrossRefGoogle Scholar
  91. 91.
    James A., Low-Choy S., Mengersen K.: Elicitator: an expert elicitation tool for regression in ecology. Environ. Model. Softw. 25(1), 129–145 (2010)CrossRefGoogle Scholar
  92. 92.
    Jeffreys H.: Theory of Probability. Oxford University Press, Oxford (1961)zbMATHGoogle Scholar
  93. 93.
    Jouini M.N., Clemen R.T.: Copula models for asggregating expert opinions. Oper. Res. 44, 444–457 (1996)zbMATHCrossRefGoogle Scholar
  94. 94.
    Kadane J.B., Schervish M.J., Seidenfeld T.: Rethinking the Foundations of Statistics. Cambridge University Press, Cambridge (1999)zbMATHGoogle Scholar
  95. 95.
    Kahneman, D., Slovic, P., Tversky, A. (eds): Judgement under Uncertainty: Heuristics and Biases. Cambridge University Press, Cambridge (1982)Google Scholar
  96. 96.
    Kallen, M.J., Cooke, R.M.: Expert aggregation with dependence. In: Bonano, E.J., Camp, A.J., Majors, M.J., Thompson, R.A. (eds.) Proabilistic Safety Assessment and Management, pp. 1287–1294. Elsevier, Amsterdam (2002)Google Scholar
  97. 97.
    Karacapilidis N.I., Pappis C.P.: A framework for group decision support systems: combining AI tools and OR techniques. Eur. J. Oper. Res. 103, 373–388 (1997)zbMATHCrossRefGoogle Scholar
  98. 98.
    Keeney R.L., Raiffa H.: Decisions with Multiple Objectives: Preferences and Value Trade-offs. Wiley, New York (1976)Google Scholar
  99. 99.
    Keith D.W.: When is it better to combine expert judgements?. Clim. Chang. 33, 111–143 (1996)CrossRefGoogle Scholar
  100. 100.
    Kelly F.S.: Arrow Impossibility Theorems. Academic Press, New York (1978)zbMATHGoogle Scholar
  101. 101.
    Kennedy M.C., O’Hagan A.: Bayesian calibration of computer models. J. R. Stat. Soc. B 63, 425–464 (2001)MathSciNetzbMATHCrossRefGoogle Scholar
  102. 102.
    Koning, J.-L.: How electronic voting can escape Arrow’s Impossibility Theorem. In: Padget, J., Neira, R., Diaz de Leon, J.L. (eds.) e-Government and e-Democracy: Progress and Challenges, pp. 138–146. Zacatenco, Mexico, Instituo Politecnico Nacional, Centro de Investigacion en Computacion, Unidad Profesional “Aldofo Lopez Mateos” (2003)Google Scholar
  103. 103.
    Kurowicka D., Cooke R.M.: Uncertainty Analysis with High Dimensional Modelling. Wiley, Chichester (2006)zbMATHCrossRefGoogle Scholar
  104. 104.
    Langford, I., Marris, C., O’Riordan, T.: Public reactions to risk: social structures, images of science and the role of trust. In: Bennett, P.G., Calman, K.C. (eds.) Risk Communication and Public Health: Policy, Science and Participation, pp. 33–50. Oxford University Press, Oxford (1999)Google Scholar
  105. 105.
    Leach, M., Scoones, I., Wynne, B. (eds): Science and Citizens. Zed Books, London (2005)Google Scholar
  106. 106.
    Leamer E.E.: Specification Searches. Wiley, Chichester (1978)zbMATHGoogle Scholar
  107. 107.
    Lehmann H.P., Goodman S.N.: Bayesian communication: a clinically significant paradigm for electronic publication. J. Am. Med. Inf. Assoc. 7(3), 254–266 (2000)Google Scholar
  108. 108.
    Lichtendahl, K.C.: Bayesian Models of Expert Forecasts. PhD Thesis. Department of Business Administration, Duke University, Durham, NC (2005)Google Scholar
  109. 109.
    Lichtendahl K.C., Winkler R.L.: Probability elicitation, scoring rules, and competition among forecasters. Manag. Sci. 53(11), 1745–1755 (2007)CrossRefGoogle Scholar
  110. 110.
    Lichtenstein, S., Fischhoff, B., Phillips, L.D.: Calibration of probabilities: the state of the art to 1980. In: Kahneman, D., Slovic, P., Tversky, A. (eds.) Judgement Under Uncertainty, pp. 306–334. Cambridge University Press, Cambridge (1982)Google Scholar
  111. 111.
    Lin S.-W., Bier V.M.: A study of expert overconfidence. Reliab. Eng. Syst. Saf. 93, 711–721 (2008)CrossRefGoogle Scholar
  112. 112.
    Lin S.-W., Cheng C.-H.: The reliability of aggregated probability judgments obtained through Cooke’s classical model. J. Model. Manag. 4(2), 149–161 (2009)MathSciNetCrossRefGoogle Scholar
  113. 113.
    Lindley D.V., Tversky A., Brown R.V.: On the reconciliation of probability judgements (with discussion). J. R. Stat. Soc. A 142, 146–180 (1979)MathSciNetzbMATHCrossRefGoogle Scholar
  114. 114.
    Lindley D.V.: The improvement of probability judgements. J. R. Stat. Soc. A 145, 117–126 (1982)MathSciNetzbMATHCrossRefGoogle Scholar
  115. 115.
    Lindley D.V.: Reconciliation of probability distributions. Oper. Res. 31(5), 866–880 (1983)MathSciNetzbMATHCrossRefGoogle Scholar
  116. 116.
    Lindley, D.V.: Reconciliation of discrete probability distributions. In: Bernardo, J.M., DeGroot, M.H., Lindley, D.V., Smith, A.F.M. (eds.) Bayesian Statistics, vol. 2, pp. 375–390. North Holland, Amsterdam (1985)Google Scholar
  117. 117.
    Linstone H.A., Turoff M.: The Delphi Method: Techniques and Applications. Addison-Wesley, London (1978)Google Scholar
  118. 118.
    Lipscomb J., Parmigiani G., Hasselblad V.: Combining expert judgment by hierarchical modeling: an application to physician staffing. Manag. Sci. 44(2), 149–161 (1998)zbMATHCrossRefGoogle Scholar
  119. 119.
    Low-Choy, S., Rousseau, J., Mengersen, K.: Poster on ‘Balancing expert consensus and diversity’. Bayesian Statistics 9. Valencia (2010)Google Scholar
  120. 120.
    Madansky, A.: Externally Bayesian groups. RM-4141-PR. RAND (1964)Google Scholar
  121. 121.
    Maule A.J.: Translating risk management knowledge: the lessons to be learned from research on the perception and communication of risk. Risk Manag. 6(2), 15–27 (2004)MathSciNetGoogle Scholar
  122. 122.
    Maule, A.J.: Risk communication and organisations. In: Starbuck, W., Hodgkinson, G. (eds.) The Oxford Handbook of Organizational Decision Making. Oxford University Press, Oxford (2008)Google Scholar
  123. 123.
    Merrick J.R.W.: Getting the right mix of experts. Decis. Anal. 5(1), 43–52 (2008)CrossRefGoogle Scholar
  124. 124.
    Merrick J.R.W.: Bayesian simulation and decision analysis: an expository survey. Decis. Anal. 6(4), 222–238 (2009)MathSciNetCrossRefGoogle Scholar
  125. 125.
    Mikkelsen, T., Galmarini, S., Bianconi, R., French, S.: ENSEMBLE: methods to reconcile disparate national long-range dispersion forecasts. RISO, Roskilde (2003)Google Scholar
  126. 126.
    Morris, C.N., Normand, S.L.: Hierarchical models for combining information and meta-analyses. In: Bernardo, J.M., Berger, J.O., Dawid, A.P., Smith, A.F.M. (eds.) Bayesian Statistics 4, pp. 321–344. Oxford, Oxford University Press (1992)Google Scholar
  127. 127.
    Morton A., Ackermann F., Belton V.: Technology-driven and model-driven approaches to group decision support: focus, research philosophy, and key concepts. Eur. J. Inf. Syst. 12(2), 110–126 (2003)CrossRefGoogle Scholar
  128. 128.
    Mumpower J.L., Stewart T.R.: Expert judgement and expert disagreement. Thinking Reason. 2(2–3), 191–211 (1996)CrossRefGoogle Scholar
  129. 129.
    Murray J.V., Goldizen A.W., O’Leary R.A., McAlpine C., Possingham H.P., Low-Choy S.: How useful is expert opinion for predicting the distribution of a species within and beyond the region of expertise? A case study using brush-tailed rock-wallabies Petrogale penicillata. J. Appl. Ecol. 46(4), 842–851 (2009)CrossRefGoogle Scholar
  130. 130.
    Newbold P., Granger C.W.J.: Experience with forecasting univariate time series and the combination of forecasts. J. R. Stat. Soc. A 137, 131–164 (1974)MathSciNetCrossRefGoogle Scholar
  131. 131.
    O’Hagan A.: Eliciting expert beliefs in substantial practical applications. Statistician 47(1), 21–35 (1998)MathSciNetGoogle Scholar
  132. 132.
    O’Hagan, A., Kennedy, M.C., Oakley, J.E.: Uncertainty analysis and other inference tools for complex computer codes. In: Bernardo, J.M., Berger, J.O., Dawid, A.P., Smith, A.F.M. (eds.) Bayesian Statistics 6, pp. 503–524. Oxford University Press, Oxford (1998)Google Scholar
  133. 133.
    O’Hagan A., Forester J.: Bayesian Statistics. Edward Arnold, London (2004)zbMATHGoogle Scholar
  134. 134.
    O’Hagan A., Buck C.E., Daneshkhah A., Eiser R., Garthwaite P.H., Jenkinson D., Oakley J.E., Rakow T.: Uncertain Judgements: Eliciting Experts’ Probabilities. Wiley, Chichester (2006)zbMATHCrossRefGoogle Scholar
  135. 135.
    O’Neill S.J., Osborn T.J., Hulme M., Lorenzoni I., Watkinson A.R.: Using expert knowledge to assess uncertainties in future polar bear populations under climate change. J. Appl. Ecol. 45(6), 1649–1659 (2008)CrossRefGoogle Scholar
  136. 136.
    Ortiz N.R., Wheeler T.A., Breeding R.J., Hora S., Meyer M.A., Keeney R.L.: Use of expert judgement in NUREG 1150. Nucl. Eng. Des. 126, 313–331 (1991)CrossRefGoogle Scholar
  137. 137.
    Phillips, L.D.: Decision conferencing. In: Edwards, W., Miles, R.F., von Winterfeldt, D. (eds.) Advances in Decision Analysis: From Foundations to Applications, pp. 375–399. Cambridge University Press, Cambridge, (2007)Google Scholar
  138. 138.
    Predd J., Osherson S., Kulkarni H.: Aggregating probabilistic forecasts from incoherent and abstaining experts. Decis. Anal. 5(4), 177–189 (2008)CrossRefGoogle Scholar
  139. 139.
    Raiffa H.: The Art and Science of Negotiation: How to Resolve Conflicts and Get the Best out of Bargaining. Belknap Press/Harvard University, Cambridge (1982)Google Scholar
  140. 140.
    Raiffa H., Richardson J., Metcalfe D.: Negotiation Analysis: the Science and Art of Collaborative Decision Making. Harvard University Press, Cambridge (2002)Google Scholar
  141. 141.
    Ranjan R., Ranjan R.: Combining probability forecasts. J. R. Stat. Soc. B 72(1), 71–91 (2010)CrossRefMathSciNetGoogle Scholar
  142. 142.
    Reagan-Cirincione P.: Combining group facilitation, decision modelling and information technology to improve the accuracy of group judgement Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology. Organ. Behav. Hum. Decis. Process 58, 246–270 (1994)CrossRefGoogle Scholar
  143. 143.
    Renn O., Webler T., Rakel H., Dienel P., Johnson B.: Public participation in decision making: a three step-procedure. Policy Sci. 26(3), 189–214 (1993)CrossRefGoogle Scholar
  144. 144.
    Renn, O., Webler, T., Wiedermann, P. (eds): Fairness and Competence in Citizen Participation: Evaluating Models and Environmental Discourse. Kluwer, Dordrecht (1995)Google Scholar
  145. 145.
    Renn O.: The role of risk communication and public dialogue for improving risk management. Risk Decis. Policy 3, 3–50 (1998)Google Scholar
  146. 146.
    Renn O.: Risk Governance. Earthscan, London (2008)CrossRefGoogle Scholar
  147. 147.
    Renooij S.: Probability elicitation for belief networks: issues to consider. Knowl. Eng. Rev. 16(3), 255–269 (2001)CrossRefGoogle Scholar
  148. 148.
    Rios Insua D., Ruggeri F.: Robust Bayesian Analysis. Springer, New York (2000)zbMATHGoogle Scholar
  149. 149.
    Rios Insua, D., French, S. (eds): e Democracy: a Group Decision and Negotiation Perspective. Group Decision and Negotiation. Springer, Dordrecht (2010)Google Scholar
  150. 150.
    Rosenthal R., DiMatteo M.R.: Meta-analysis: recent developments in quantitative methods for literature reviews. Annu. Rev. Psychol. 52, 59–82 (2001)CrossRefGoogle Scholar
  151. 151.
    Rosqvist T.: Bayesian aggregation of experts’ judgements on failure intensity. Reliab. Eng. Syst. Saf. 70(3), 283–289 (2000)CrossRefGoogle Scholar
  152. 152.
    Rougier J.C., Guillas S., Maute A., Richmond A.D.: Expert knowledge and multivariate emulation: the thermospher-ionosphere electrodynamics general circulation model (TIE-GCM). Technometrics 51(4), 414–424 (2009)CrossRefMathSciNetGoogle Scholar
  153. 153.
    Rowe G., Wright G.: The Delphi technique as a forecasting tool: issues and analysis. Int. J. Forecast. 15, 353–375 (1999)CrossRefGoogle Scholar
  154. 154.
    Savage L.J.: The Foundations of Statistics. Dover, New York (1972)zbMATHGoogle Scholar
  155. 155.
    Scapolo F., Miles I.: Eliciting experts’ knowledge: a comparison of two methods. Technol. Forecast. Soc. Chang. 73(6), 679–704 (2006)CrossRefGoogle Scholar
  156. 156.
    Shanteau, J.: Expert Judgment and Financial Decision Making. In: Green, B. (ed.) Risky Business: Risk Behavior and Risk Management. Stockholm University, Stockholm (1995)Google Scholar
  157. 157.
    Skjong, R., Wentworth, B.H.: Expert Judgement and Risk Perception. In: Proceedings of the Eleventh (2001) International Offshore and Polar Engineering Conference, Stavanger, Norway, The International Society of Offshore and Polar Engineers (2001)Google Scholar
  158. 158.
    Slovic, P.: Trust, emotion, sex, politics and science: surveying the risk-assessment battlefield. In: Bazerman, M., Messick, D., Tenbrunsel, A., Wade-Benzoni, K. (eds.) Environment, Ethics and Behaviour. New Lexington Press, San Francisco (1997)Google Scholar
  159. 159.
    Slovic P.: Perceptions of Risk. Earthscan Library, London (2001)Google Scholar
  160. 160.
    Smith J.Q., Faria A.: Bayesian Poisson models for the graphical combination of dependent expert information. J. R. Stat. Soc. B 62(3), 525–544 (2000)MathSciNetzbMATHCrossRefGoogle Scholar
  161. 161.
    Smith J.Q.: Bayesian Decision Analysis: Principles and Practice. Cambridge University Press, Cambridge (2010)zbMATHGoogle Scholar
  162. 162.
    Smith L.: Chaos: A Very Short Introduction. Oxford University Press, Oxford (2007)Google Scholar
  163. 163.
    Stanley T.D.: Wheat from chaff: meta-analysis as quantitative literature review. J. Econ. Perspect. 15(3), 131–151 (2001)CrossRefGoogle Scholar
  164. 164.
    Suantak L., Bolger F., Ferrell W.R.: The hard-easy effect in subjective probability calibration. Organ. Behav. Human Decis. Process. 67, 201–221 (1996)CrossRefGoogle Scholar
  165. 165.
    Surowiecki J.: The Wisdom of Crowds. Random House, New York (2004)Google Scholar
  166. 166.
    Sutton A.J., Abrams K.R.: Bayesian methods in meta-analysis and evidence synthesis. Stat. Methods Med. Res. 10(4), 277–303 (2001)zbMATHCrossRefGoogle Scholar
  167. 167.
    Szwed, P.S., van Dorp, J.R.: A Bayesian Model for Rare Event Risk Assessment Using Expert Judgement About Paired Scenario Comparisons. ASEM National Conference (2002)Google Scholar
  168. 168.
    Szwed P.S., van Dorp J.R., Merrick J.R.W., Mazzuchi T.A., Singh A.: A Bayesian paired comparison approach for relative accident probability assessment with covariate information. Eur. J. Oper. Res. 169(1), 157–177 (2006)zbMATHCrossRefGoogle Scholar
  169. 169.
    Taylor A.D.: Social Choice and the Mathematics of Manipulation. Cambridge University Press, Cambridge (2005)zbMATHCrossRefGoogle Scholar
  170. 170.
    Turner R.M., Speigelhalter D.J., Smith G.C.S., Thompson S.G.: Bias modelling in evidence synthesis. J. R. Stat. Soc. A 172(1), 21–47 (2009)CrossRefGoogle Scholar
  171. 171.
    Winkler R.L.: Combining probability distributions from dependent information sources. Manag. Sci. 27, 479–488 (1981)zbMATHCrossRefGoogle Scholar
  172. 172.
    Wiper, M.W.: Calibration and use of expert probability judgements, PhD. School of Computer Studies, University of Leeds, Leeds (1990)Google Scholar
  173. 173.
    Wiper M.W., French S., Cooke R.M.: Hypothesis test based calibration scores. Statistician 43, 231–236 (1994)CrossRefGoogle Scholar
  174. 174.
    Wiper M.W., French S.: Combining experts’ opinions using a normal-Wishart model. J. Forecast. 14, 25–34 (1995)CrossRefGoogle Scholar
  175. 175.
    Wisse B., Bedford T., Quigley J.: Expert judgement combination using moment methods. Reliab. Eng. Syst. Saf. 93, 675–686 (2007)CrossRefGoogle Scholar
  176. 176.
    Wright, G., Ayton, P. (eds): Subjective Probability. Wiley, Chichester (1994)Google Scholar
  177. 177.
    Zacharakis A.L., Shepherd D.A.: The nature of information and overconfidence on venture capitalist’s decision making. J. Bus. Ventur. 16, 311–332 (2001)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag 2011

Authors and Affiliations

  1. 1.Manchester Business SchoolManchesterUK

Personalised recommendations