Abstract
The National Defense University's study of climate change to the year 2000 was based largely on the judgments of the members of two expert panels. Although the study has been widely distributed and apparently read by policy makers in the U.S. and abroad, the method of eliciting and analyzing expert judgment has not been critically reviewed. This paper uses the literature on judgment and subjective probability to evaluate the expert judgment methods used in the study.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Armstrong, J. Scott: 1978, Long-Range Forecasting: From Crystal Ball to Computer, Wiley, New York.
Brown, L. R.: 1970, Seeds of Change, Praeger, New York.
Clemen, R. T. and Winkler, R. L.: 1983, ‘Limits for the Precision and Value of Information from Dependent Sources’, Unpublished manuscript, University of Indiana, Bloomington, Indiana.
Congressional Research Service: 1976, A Primer on Climatic Variation and Change, U.S. Government Printing Office, Washington.
Council on Environmental Quality: 1980, ‘The Global 2000 Report to the President: Entering the Twenty-First Century’, Washington, D.C. U.S. Government Printing Office.
Daan, H. and Murphy, A. H.: 1982, ‘Subjective Probability Forecasting in The Netherlands: Some Operational and Experimental Results’, Meteorol. Rund. 35, 99–112.
Einhorn, H. J. and Hogarth, R. M.: 1982, ‘Prediction, Diagnosis, and Causal Thinking in Forecasting’, Journal of Forecasting 1, 23–36.
Einhorn, H. J. and Hogarth, R. M.: 1981, ‘Behavioral Decision Theory: Processes of Judgment and Choice’, Annual Review of Psychology 32, 53–88.
Einhorn, H. J., Hogarth, R. M., and Klempner, E.: 1977, ‘Quality of Group Judgment’, Psychological Bulletin 84, 158–172.
Fischer, G. W.: 1981, ‘When Oracles Fail - A Comparison of Four Procedures for Aggregating Subjective Probability Forecasts’, Organizational Behavior and Human Performance 28, 96–110.
Fischhoff, B.: 1982, lsDebiasing’, in D. Kahneman, P. Slovic, and A. Tversky (eds.) Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press, New York.
Fischhoff, B. and MacGregor, D.: 1982, ‘Subjective Confidence in Forecasts’, Journal of Forecasting 1, 155–172.
Glantz, M. H., Robinson, J., and Krenz, M.: 1982, ‘Improving the Science of Climate-Related Impact Studies: A Review of Past Experience’, in William Clark (ed.), Carbon Dioxide Review: 1982, Oxford University Press, New York.
Guilford, J. P.: 1954, Psychometric Methods, McGraw Hill, New York, p. 25.
Hammond, K. R., Anderson, B. F., Sutherland, J., and Marvin, B.: 1982, ‘Improving Scientists’ Judgments of Risk, University of Colorado at Boulder, Center for Research on Judgment and Policy, Report No. 239.
Hammond, K. R., McClelland, G. H., and Mumpower, J.: 1980, Human Judgment and Decision Making: Theories, Methods and Procedures, Praeger, New York.
Hammond, K. R., Rohrbaugh, J., Mumpower, J., and Adelman, L.: 1977, ‘Social Judgment Theory: Applications in Policy Formation’, in M. F. Kaplan and S. Schwartz (eds.), Human Judgment and Decision Processes in Applied Settings, Academic Press, New York.
Hammond, K. R., Stewart, T. R., Brehmer, B., and Steinmann, D. O.: 1975, ‘Social Judgment Theory’, in M. F. Kaplan and S. Schwartz (eds.), Human Judgment and Decision Processes, Academic Press, New York.
Hogarth, R. M.: 1980, Judgment and Choice: The Psychology of Decision, Wiley, Chichester, England.
Hogarth, R. M.: 1977, ‘Methods for Aggregating Opinions’, in H. Jungermann and G. de Zeeuw (eds.), Decision Making and Change in Human Affairs, D. Reidel Publ. Co., Dordrecht, Holland, pp. 231–255.
Hogarth, R. M.: 1975, ‘Cognitive Processes and the Assessment of Subjective Probability Distributions’, J. Amer. Statist. Assoc. 70, 271–289.
Hopkins, R. F. and Puchala, D. J. (eds.): 1978, The Global Political Economy of Food, U. of Wisconsin Press, Madison, Wisconsin, p. 2.
Koriat, A., Lichtenstein, S., and Fischhoff, B.: 1980, ‘Reasons for Confidence’, J. Experimental Psych.: Human Learning and Memory 6, 107–118.
Kraemer, R. S.: 1978, ‘Meeting Reviews: Session on Climatic Futures at the Annual Meeting of the AAAS, 17 February, 1978’, Bulletin of the American Meteorological Society 59(7), 822–23.
Lichtenstein, S., Fischhoff, B., and Phillips, E. D.: 1982, ‘Calibration of Probabilities: The State of the Art to 1980’, in D. Kahneman, P. Slovic, and A. Tversky (eds.), Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press, New York.
Linstone, H. A. and Turoff, M.: 1975, The Delphi Method: Techniques and Applications, Addison-Wesley, Reading, Mass.
McKay, G. A. and Williams, G. D. V.: 1982, ‘Crop Yields and Climate Change to the Year 2000, Volume I (Book Review)’, Bulletin of the American Meteorological Society 63, 427.
Murphy, A. H. and Brown, B. G.: 1984, ‘A Comparative Evaluation of Objective and Subjective Weather Forecasts in the United States’, Journal of Forecasting 3.
Murphy, A. H. and Daan, H.: 1984, ‘Impacts of Feedback and Experience on the Quality of Subjective Probability Forecasts: Comparison of Results from the First and Second Years of the Zierikzee Experiment’, Monthly Weather Rev. 112, 413–423.
Murphy, A. H. and Winkler, R. L.: 1974, ‘Credible Interval Temperature Forecasting: Some Experimental Results’, Monthly Weather Rev. 102, 784–794.
National Academy of Sciences: 1975, Environmental Impact of Stratospheric Flight. National Academy of Sciences, Washington, D.C.
NDU: 1983, The World Grain Economy and Climate Change to the Year 2000: Implications for Policy. National Defense University Press, Washington, D.C.
NDU: 1980, Crop Yields and Climate Change to the Year 2000, National Defense University, Washington, D.C.
NDU: 1978, Climate Change to the Year 2000, National Defense University, Washington, D.C.
Newell, A. and Simon, H. A.: 1972, Human Problem Solving, Prentice-Hall, Inc., Englewood Cliffs, N.J.
Nunnally, J. C: 1978, Psychometric Theory, 2nd ed., McGraw Hill, New York.
Raiffa, H.; 1968, Decision Analysis: Introductory Lectures on Choices under Uncertainty, Addison-Wesley Publishing Co., Reading, Massachusetts.
Rohrbaugh, J.: 1979, ‘Improving the Quality of Group Judgment: Social Judgment Analysis and the Delphi Technique’, Organizational Behavior and Human Performance 24, 73–92.
Sackman, H.: 1975, Delphi Critique, D.C. Heath and Company, Lexington, Mass.
Schuman, H. and Presser, S.: 1981, Question, and Answers in Attitude Surveys. Academic Press, New York.
Science News: 1978, ‘The 25-Year Forecast: Group Prediction’, 113(8), 116.
Seaver, D. A.: 1976, ‘Assessment of Group Preferences and Group Uncertainty for Decision Making’, Technical Report SSRI 76–4, Social Science Research Institute, University of Southern California, Los Angeles, California.
Seaver, D. A., von Winterfeldt, D., and Edwards, W.: 1978, ‘Eliciting Subjective Probability Distributions on Continuous Varibales’, Organizational Behavior and Human Performance 21, 379–391.
Sellers, W. D.: 1979, ‘Climate Change to the Year 2000: A Book Review’, Bulletin of the American Meteorological Society 60(6), 686.
Slovic, P., Fischhoff, B., and Lichtenstein, S.: 1977, ‘Behavioral Decision Theory’, Annual Review of Psychology 28, 1–39.
Slovic, P. and Lichtenstein, S.: 1971, ‘Comparison of Bayesian and Regression Approaches to the Study of Information Processing in Judgment’, Organizational Behavior and Human Performance 6, 649–744.
Spetzler, C. S. and Stael von Holstein, C.-A.S.: 1975, ‘Probability Encoding in Decision Analysis’, Management Sci. 22, 340–358.
Stewart, T. R.: 1984, ‘Judgment and Forecasting: Methodological Implications of Judgment Research’. Paper prepared for the Conference on Forecasting in the Social and Natural Sciences, Boulder, CO, June 10–13.
Tversky, A. and Kahneman, D.: 1981, ‘The Framing of Decisions and the Rationality of Choice’, Science 211, 453–458.
Tversky, A. and Kahneman, D.: 1974, ‘Judgment under Uncertainty: Heuristics and Biases’, Science 185, 1124–1131.
Tversky, A. and Kahneman, D.: 1971, ‘The Belief in the Law of Small Numbers’, Psychological Bulletin 76, 105–10.
U.N. FAO: 1979, ‘Scanning the Future for Climate Change’, CERES 12, 7.
U.S. Central Intelligence Agency: 1974, A Study of Climatological Research as It Pertains to Intelligence Problems, Document Expediting Project, Library of Congress, Washington, D.C.
Wallsten, T.S. and Budescu, D.V.: 1983, ‘Encoding Subjective Probabilities: A Psychological and Psychometric Review’, Management Science 29, 151–173.
Winkler, R.L.: 1968, ‘The Consensus of Subjective Probability Distributions’, Management Science B 15, 61–75.
Winkler, R. L. and Murphy, A.H.: 1979, ‘The Use of Probabilities in Forecasts of Maximum and Minimum Temperatures’, Meteorological Magazine 108, 317–329.
Author information
Authors and Affiliations
Additional information
The National Center for Atmospheric Research is sponsored by the National Science Foundation.
Rights and permissions
About this article
Cite this article
Stewart, T.R., Glantz, M.H. Expert judgment and climate forecasting: A methodological critique of “climate change to the year 2000”. Climatic Change 7, 159–183 (1985). https://doi.org/10.1007/BF00140504
Received:
Revised:
Issue Date:
DOI: https://doi.org/10.1007/BF00140504