Risk Analysis

, Volume 19, Issue 2, pp 187–203 | Cite as

Combining Probability Distributions From Experts in Risk Analysis

  • Robert T. Clemen
  • Robert L. Winkler
Article

Abstract

This paper concerns the combination of experts' probability distributions in risk analysis, discussing a variety of combination methods and attempting to highlight the important conceptual and practical issues to be considered in designing a combination process in practice. The role of experts is important because their judgments can provide valuable information, particularly in view of the limited availability of “hard data” regarding many important uncertainties in risk analysis. Because uncertainties are represented in terms of probability distributions in probabilistic risk analysis (PRA), we consider expert information in terms of probability distributions. The motivation for the use of multiple experts is simply the desire to obtain as much information as possible. Combining experts' probability distributions summarizes the accumulated information for risk analysts and decision-makers. Procedures for combining probability distributions are often compartmentalized as mathematical aggregation methods or behavioral approaches, and we discuss both categories. However, an overall aggregation process could involve both mathematical and behavioral aspects, and no single process is best in all circumstances. An understanding of the pros and cons of different methods and the key issues to consider is valuable in the design of a combination process for a specific PRA. The output, a “combined probability distribution,” can ideally be viewed as representing a summary of the current state of expert opinion regarding the uncertainty of interest.

Combining probabilities expert judgment probability assessment 

REFERENCES

  1. Argote, L., Seabright, M. A., & Dyer, L. (1986). Individual versus group use of base-rate and individuating information. Organizational Behavior and Human Decision Processes 3865–75.Google Scholar
  2. Armstrong, J. S., Denniston, W. B., & Gordon, M. M. (1975). The use of the decomposition principle in making judgments. Organizational Behavior and Human Performance 14257–263.Google Scholar
  3. Arrow, K. J. (1951). Social Choice and Individual Values. New York: Wiley.Google Scholar
  4. Aumann, R. J. (1976). Agreeing to Disagree. Annals of Statistics 41236–1239.Google Scholar
  5. Bacharach, M. (1979). Normal Bayesian dialogues. Journal of the American Statistical Association 74837–846.Google Scholar
  6. Bates, J. M., & Granger, C. W. J. (1969). The combination of forecasts. Operational Research Quarterly 20451–468.Google Scholar
  7. Bonano, E. J., Hora, S. C., Keeney, R. L., & von Winterfeldt, D. (1990). Elicitation and Use of Expert Judgment in Performance Assessment for High-Level Radioactive Waste Repositories. NUREG/CR-5411. Washington, DC: Nuclear Regulatory Commission.Google Scholar
  8. Bonduelle, Y. (1987). Aggregating Expert Opinions by Resolving Sources of Disagreement. PhD Dissertation, Stanford University.Google Scholar
  9. Brockhoff, K. (1975). The performance of forecasting groups in computer dialogue and face-to-face discussion. In H. Linstone & M. Turoff (Eds.), The Delphi Method: Techniques and Applications. Reading, MA: Addison-Wesley.Google Scholar
  10. Bunn, D. W. (1975). A Bayesian approach to the linear combination of forecasts. Operational Research Quarterly 26325–329.Google Scholar
  11. Bunn, D. W. (1988). Combining forecasts. European Journal of Operational Research 33223–229.Google Scholar
  12. Bunn, D., & Wright, G. (1991). Interaction of judgmental and statistical forecasting methods: Issues and Analysis. Management Science 37501–518.Google Scholar
  13. Chandrasekharan, R., Moriarty, M. M., & Wright, G. P. (1994). Testing for unreliable estimators and insignificant forecasts in combined forecasts. Journal of Forecasting 13611–624.Google Scholar
  14. Chhibber, S., & Apostolakis, G. (1993). Some approximations useful to the use of dependent information sources. Reliability Engineering and System Safety 4267–86.Google Scholar
  15. Chhibber, S., Apostolakis, G., & Okrent, D. (1992). A taxonomy of the use of expert judgments in safety studies. Reliability Engineering & System Safety 3827–45.Google Scholar
  16. Clemen, R. T. (1985). Extraneous expert information. Journal of Forecasting 4329–348.Google Scholar
  17. Clemen, R. T. (1986). Calibration and the aggregation of probabilities. Management Science 32312–314.Google Scholar
  18. Clemen, R. T. (1989). Combining forecasts: A review of annotated bibliography. International Journal of Forecasting 5559–583.Google Scholar
  19. Clemen, R. T., Jones, S. K., & Winkler, R. L. (1996). Aggregating forecasts: An empirical evaluation of some Bayesian methods. In D. Berry, K. Chaloner, & J. Geweke (Eds.), Bayesian Statistics and Econometrics: Essays in Honor of Arnold Zellner(pp 3–13). New York: Wiley.Google Scholar
  20. Clemen, R. T., & Murphy, A. H. (1986). Objective and subjective precipitation probability forecasts: Statistical analysis of some interrelationships. Weather and Forecasting 156–65.Google Scholar
  21. Clemen, R. T., & Reilly, T. (1999). Correlations and copulas for decision and risk analysis. Management Science 45208–224.Google Scholar
  22. Clemen, R. T., & Winkler, R. L. (1985). Limits for the precision and value of information from dependent sources. Operations Research 33427–442.Google Scholar
  23. Clemen, R. T., & Winkler, R. L. (1987). Calibrating and combining precipitation probability forecasts. In R. Viertl (Ed.), Probability and Bayesian Statistics(pp. 97–110). New York: Plenum.Google Scholar
  24. Clemen, R. T., & Winkler, R. L. (1990). Unanimity and compromise among probability forecasters. Management Science 36767–779.Google Scholar
  25. Clemen, R. T., & Winkler, R. L. (1993). Aggregating point estimates: A flexible modeling approach. Management Science 39501–515.Google Scholar
  26. Cooke, R. M. (1991). Experts in Uncertainty: Opinion and Subjective Probability in Science. New York: Oxford University Press.Google Scholar
  27. Dalkey, N. C. (1969). The Delphi method: An experimental study of group opinions. Report No. RM-5888-PR. The Rand Corporation.Google Scholar
  28. Dalkey, N. C., & Brown, B. (1971). Comparison of group judgment techniques with short-range predictions and almanac questions. Report No. R-678-ARPA. The RAND Corporation.Google Scholar
  29. Dall'Aglio, G., Kotz, S., & Salinetti, G. (1991). Advances in Probability Distributions with Given Marginals: Beyond the CopulasDordrecht, Netherlands: Kluwer.Google Scholar
  30. Davis, J. (1992). Somecompelling intuitions about group consensus decisions, theoretical and empirical research, and interpersonal aggregation phenomena: Selected examples, 1950- 1990. Organizational Behavior and Human Decision Processes 523–38.CrossRefGoogle Scholar
  31. Dawes, R. M., Faust, D., & Meehl, P. A. (1989). Clinical versus actuarial judgment. Science 2431668–1673.PubMedGoogle Scholar
  32. de Finetti, B. (1937). La Pre´ vision: Ses Lois Logiques, Ses Sources Subjectives. Annales De L'Institut Henri Poincare´7 Google Scholar
  33. Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1975). Group Techniques for Program Planning. Glenview, IL: Scott Foresman.Google Scholar
  34. Einhorn, H. J., Hogarth, R. M., & Klempner, E. (1977). Quality of group judgment. Psychological Bulletin 84158- 172.Google Scholar
  35. EPRI (1986). Seismic Hazard Methodology for the Central and Eastern United States.Vol.1: Methodology. NP-4/26. Palo Alto, CA: Electric Power Research Institute.Google Scholar
  36. Ferrell, W. R. (1985) Combining individual judgments. In G. Wright (Ed.), Behavioral Decision Making(pp. 111- 145). New York: Plenum.Google Scholar
  37. Fischer, G. (1975). An experimental study of four procedures for aggregating subjective probability assessments. Technical report 75-7. Decisions and Designs, Inc.Google Scholar
  38. Flores, B. E., & White, E. M. (1989). Subjective vs. objective combining of forecasts: An experiment. Journal of Forecasting 8331–341.Google Scholar
  39. French, S. (1981). Consensus of opinion. European Journal of Operational Research 7332–340.Google Scholar
  40. French, S. (1985). Group consensus probability distributions: A critical survey. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), Bayesian Statistics 2(pp. 183–197). Amsterdam: North-Holland.Google Scholar
  41. Gelfand, A. E., Mallick, B. K., & Dey, D. K. (1995). Modeling expert opinion rising as a partial probabilistic specification. Journal of the Ameican Statistical Association 90598–604.Google Scholar
  42. Genest, C. (1984). Pooling operators with the marginalization property. Canadian Journal of Statistics 12153–163.Google Scholar
  43. Genest, C., & McConway, K. J. (1990). Allocating the weights in the linear opinion pool. Journal of Forecasting 953–73.Google Scholar
  44. Genest, C., & Schervish, M. J. (1985). Modeling expert judgments for Bayesian updating. Annals of Statistics 131198–1212.Google Scholar
  45. Genest, C., & Zidek, J. V. (1986). Combining probability distributions. A Critique and annotated bibliography. Statistical Science 1114–148.Google Scholar
  46. Gigone, D., & Hastie, R. (1997). Proper analysis of the accuracy of group judgments. Psychological Bulletin 121149–167.Google Scholar
  47. Gokhale, D. V., & Press, S. J. (1982). Assessment of a prior distribution for the correlation coefficient in a bivariate normal distribution. Journal of the Royal Statistical Society, Series A 145237–249.Google Scholar
  48. Goodman, B. (1972). Action selection and likelihood estimation by individuals and groups. Organizational Behavior and Human Performance 7121–141.Google Scholar
  49. Gough, R. (1975). The effects of group format on aggregate subjective probability distributions. In Utility, Probability, and Human Decision Making. Dordrecht, Netherlands: Reidel.Google Scholar
  50. Gustafson, D. H., Shukla, R. U., Delbecq, A., & Walster, G. W. (1973).Acomparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups. Organizational Behavior and Human Performance 9280–291.Google Scholar
  51. Hastie, R. (1986). Review essay: Experimental evidence on group accuracy. In B. Grofman & G. Owen (Eds.), Information Pooling and Group Decision Making: Proceedings of the Second University of California, Irvine, Conference on Political Economy. Greenwich, CT: JAI Press.Google Scholar
  52. Hammitt, J. K., & Shlyakhter, A. I. (1999). The expected value of information and the probability of surprise. Risk Analysis 19135–152.Google Scholar
  53. Hill, G. W. (1982). Group vs. individual performance: Are N 1 heads better than one? Psychological Bulletin 91517–539.Google Scholar
  54. Hogarth, R. M. (1977). Methods for aggregating opinions. In H. Jungermann & G. DeZeeuw (Eds.), Decision Making and Change in Human Affairs(pp. 231–255). Dordrecht, Netherlands: Reidel.Google Scholar
  55. Hogarth, R. M. (1987). Judgment and Choice: 2nd Ed. Chichester, England: Wiley.Google Scholar
  56. Hora, S. C. (1992). Acquisition of expert judgment: Examples from risk assessment. Journal of Energy Engineering 118136–148.Google Scholar
  57. Hora, S. C., Dodd, N. G., & Hora, J. A. (1993). The use of decomposition in probability assessments on continuous variables. Journal of Behavioral Decision Making 6133–147.Google Scholar
  58. Hora, S. C., & Iman, R. L. (1989). Expert opinion in risk analysis: The NUREG-1150 methodology. Nuclear Science and Engineering 102323–331.Google Scholar
  59. Innami, I. (1994). The quality of group decisions, group verbal behavior, and intervention. Organizational Behavior and Human Decision Processes 60409–430.Google Scholar
  60. Janis, I. L. (1982). Groupthink: Psychological Studies of Policy Decisions and Fiascoes, 2nd Ed. Boston: Houghton Mifflin.Google Scholar
  61. Janis, I. L., & Mann, L. (1977). Decision Making. New York: Free Press.Google Scholar
  62. Jouini, M. N., & Clemen, R. T. (1996). Copula models for aggregating expert opinions. Operations Research 44444–457.Google Scholar
  63. Kahneman, D., & Tversky, A. (1972). Subjective probability: A judgment of representativeness. Cognitive Psychology 3430–454.Google Scholar
  64. Kaplan, S. (1990). ‘Expert information’ vs ‘expert opinions’: Another approach to the problem of eliciting/combining/using expert knowledge in PRA. Journal of Reliability Engineering and System Safety, 39.Google Scholar
  65. Keeney, R. L., & von Winterfeldt, D. (1989). On the uses of expert judgment on complex technical problems. IEEE Transactions on Engineering Management 3683–86.Google Scholar
  66. Keeney, R. L., & von Winterfeldt, D. (1991). Eliciting probabilities from experts in complex technical problems. IEEE Transactions on Engineering Management 38191–201.Google Scholar
  67. Lawrence, M. J., Edmundson, R. H., & O'Connor, M. J. (1986). The accuracy of combining judgmental and statistical forecasts. Management Science 321521–1532.Google Scholar
  68. Lindley, D. V. (1983). Reconciliation of probability distributions. Operations Research 31866–880.Google Scholar
  69. Lindley, D. V. (1985). Reconciliation of discrete probability distributions. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), Bayesian Statistics 2(pp. 375- 390). Amsterdam: North-Holland.Google Scholar
  70. Linstone, H. A., & Turoff, M. (1975). The Delphi Method: Techniques and Applications. Reading, MA: Addison-Wesley.Google Scholar
  71. Lipscomb, J., Parmigiani, G., & Hasselblad, V. (1998). Combining expert judgment by hierarchical modeling: An application to physician staffing. Management Science 44149–161.Google Scholar
  72. Lock, A. (1987). Integrating group judgments in subjective forecasts. In G. Wright & P. Ayton (Eds.), Judgmental Forecasting(pp. 109–127). Chichester, England: Wiley.Google Scholar
  73. MacGregor, D., Lichtenstein, S., & Slovic, P. (1988). Structuring knowledge retrieval: An analysis of decomposing quantitative judgments. Organizational Behavior and Human Decision Processes 42303–323.Google Scholar
  74. Makridakis, S., and Winkler, R. L. (1983). Averages of forecasts: Some empirical results. Management Science 29987–996.Google Scholar
  75. Mendel, M. B., & Sheridan, T. B. (1989). Filtering information from human experts. IEEE Transactions on Systems, Man, and Cybernetics 366–16.Google Scholar
  76. Merkhofer, M. W. (1987). Quantifying judgmental uncertainty: Methodology, experience, and insights. IEEE Transactions on Systems, Man, and Cybernetics 17741–752.Google Scholar
  77. Morgan, M. G., & Henrion, M. (1990). Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge, MA: Cambridge University Press.Google Scholar
  78. Morgan, M. G., & Keith, D. W. (1995). 3 Subjective judgments by climate experts. Environmental Science and Technology 29468–476.Google Scholar
  79. Morris, P. A. (1974). Decision analysis expert use. Management Science 201233–1241.Google Scholar
  80. Morris, P. A. (1977). Combining expert judgments: A Bayesian approach. Management Science 23679–693.Google Scholar
  81. Morris, P. A. (1983). An axiomatic approach to expert resolution. Management Science 2924–32.Google Scholar
  82. Mosleh, A., Bier, V. M., & Apostolakis, G. (1987). A critique of current practice for the use of expert opinions in probabilistic risk assessment. Reliability Engineering and System Safety 2063–85.Google Scholar
  83. Myers, D. G., & Lamm, H. (1975). The polarizing effect of group discussion. American Scientist 63297–303.PubMedGoogle Scholar
  84. Newbold, P., & Granger, C. W. J. (1974). Experience with forecasting univariate time series and the combination of forecasts. Journal of the Royal Staistical Society, Series A 137131–149.Google Scholar
  85. Otway, H., & von Winterfeldt, D. (1992). Expert judgment in risk analysis and management: Process, context, and pitfalls. Risk Analysis 1283–93.PubMedGoogle Scholar
  86. Parente´, F. J., & Anderson-Parente´, J. K. (1987). Delphi inquiry systems. In G. Wright & P. Ayton (Eds.), Judgmental Forecasting(pp. 129–156). Chichester, England: Wiley.Google Scholar
  87. Park, W. (1990). A review of research on groupthink. Journal of Behavioral Decision Making 3229–245.Google Scholar
  88. Phillips, L. D. (1984). A theory of requisite decision models. Acta Psychologica 5629–48.CrossRefGoogle Scholar
  89. Phillips, L. D. (1987). On the adequacy of judgmental forecasts. In G. Wright & P. Ayton (Eds.), Judgmental Forecasting(pp. 11–30). Chichester, England: Wiley.Google Scholar
  90. Phillips, L. D., & Phillips, M. C. (1990). Facilitated work groups: Theory and practice. Unpublished manuscript, London School of Economics and Political Science.Google Scholar
  91. Plous, S. (1993). The Psychology of Judgment and Decision Making. New York: McGraw-Hill.Google Scholar
  92. Ravinder, H. V., Kleinmuntz, D. N., & Dyer, J. S. (1988). The reliability of subjective probabilities obtained through decomposition. Management Science 34186–199.Google Scholar
  93. Reagan-Cirincione, P. (1994). Improving the accuracy of group judgment; A process intervention combining group facilitation, social judgment analysis, and information technology. Organizational Behavior and Human Decision Processes 58246–270.Google Scholar
  94. Rohrbaugh, J. (1979). Improving the quality of group judgment: Social judgment analysis and the Delphi technique. Organizational Behavior and Human Performance 2473–92.CrossRefGoogle Scholar
  95. Savage, L. J. (1954). The Foundations of Statistics. New York: Wiley.Google Scholar
  96. Schmittlein, D. C., Kim, J., & Morrison, D. G. (1990). Combining forecasts: Operational adjustments to theoretically optimal rules. Management Science 361044–1056.Google Scholar
  97. Seaver, D. A. (1978). Assessing probability with multiple individuals: Group interaction versus mathematical aggregation(Report No. 78-3). Social Science Research Institute, University of Southern California.Google Scholar
  98. Shlyakhter, A. I. (1994). Improved framework for uncertainty analysis: Accounting for unsuspected errors. Risk Analysis 14441–447.Google Scholar
  99. Shlyakhter, A. I., Kammen, D. M., Brodio, C. L., & Wilson, R. (1994). Quantifying the credibility of energy projections from trends in past data: The U.S. energy sector. Energy Policy 22119–130.Google Scholar
  100. Sniezek, J. A. (1989). An examination of group process in judgmental forecasting. International Journal of Forecasting 5171–178.Google Scholar
  101. Sniezek, J. (1992). Groups under uncertainty: An examination of confidence in group decision making. Organizational Behavior and Human Decision Processes 52124–155.Google Scholar
  102. Sniezek, J. A., & Henry, R. A. (1989). Accuracy and confidence in group judgment. Organizational Behavior and Human Decision Processes 431–28.CrossRefGoogle Scholar
  103. Sniezek, J. A., & Henry, R. A. (1990). Revision, weighting, and commitment in consensus group judgment. Organizational Behavior and Human Decision Processes 4566–84.CrossRefGoogle Scholar
  104. Stae¨ l von Holstein, C.-A. S. (1972). Probabilistic forecasting: An experiment related to the stock market. Organizational Behavior and Human Performance 8139–158.Google Scholar
  105. Stone, M. (1961). The opinion pool. Annals of Mathematical Statistics 321339–1342.Google Scholar
  106. Tindale, R. S., Sheffey, S., & Filkins, J. (1990). Conjunction errors by individuals and groups. Paper presented at the annual meeting of the Society for Judgment and Decision Making, New Orleans, LA.Google Scholar
  107. Uecker, W. C. (1982). The quality of group performance in simpli-fied information evaluation. Journal of Accounting Research 20388–402.Google Scholar
  108. West, M. (1992). Modelling agent forecast distributions. Journal of the Royal Statistical Society B 54553–567.Google Scholar
  109. West, M., & Crosse, J. (1992). Modelling probabilistic agent opinion. Journal of the Royal Statistical Society B 54285–299.Google Scholar
  110. Winkler, R. L. (1968). The consensus of subjective probability distributions. Management Science 15361–375.Google Scholar
  111. Winkler, R. L. (1981). Combining probability distributions from dependent information sources. Management Science 27479–488.Google Scholar
  112. Winkler, R. L., & Clemen, R. T. (1992). Sensitivity of weights in combining forecasts. Operations Research 40609–614.Google Scholar
  113. Winkler, R. L., & Makridakis, S. (1983). The combination of forecasts. Journal of the Royal Statistical Society, Series A 146150–157.Google Scholar
  114. Winkler, R. L., & Poses, R. M. (1993). Evaluating and combining physicians' probabilities of survival in an intensive care unit. Management Science 391526–1543.Google Scholar
  115. Winkler, R. L., Wallsten, T. S., Whitfield, R. G., Richmond, H. M., Hayes, S. R., & Rosenbaum, A. S. (1995). An assessment of the risk of chronic lung injury attributable to longterm ozone exposure. Operations Research 4319–28.Google Scholar
  116. Wright, G., Saunders, C., & Ayton, P. (1988). The consistency, coherence and calibration of holistic, decomposed, and recomposed judgmental probability forecasts. Journal of Forecasting 7185–199.Google Scholar
  117. Wu, J. S., Apostolakis, G., & Okrent, D. (1990). Uncertainties in system analysis: Probabilistic versus nonprobabilistic theories. Reliability Engineering & System Safety 30163–181.Google Scholar

Copyright information

© Society for Risk Analysis 1999

Authors and Affiliations

  • Robert T. Clemen
    • 1
  • Robert L. Winkler
    • 2
  1. 1.Fuqua School of Business, Duke UniversityDurham
  2. 2.Fuqua School of Business, Duke UniversityDurham

Personalised recommendations