Climatic Change

, Volume 7, Issue 2, pp 159–183 | Cite as

Expert judgment and climate forecasting: A methodological critique of “climate change to the year 2000”

  • Thomas R. Stewart
  • Michael H. Glantz
Article

Abstract

The National Defense University's study of climate change to the year 2000 was based largely on the judgments of the members of two expert panels. Although the study has been widely distributed and apparently read by policy makers in the U.S. and abroad, the method of eliciting and analyzing expert judgment has not been critically reviewed. This paper uses the literature on judgment and subjective probability to evaluate the expert judgment methods used in the study.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Armstrong, J. Scott: 1978, Long-Range Forecasting: From Crystal Ball to Computer, Wiley, New York.Google Scholar
  2. Brown, L. R.: 1970, Seeds of Change, Praeger, New York.Google Scholar
  3. Clemen, R. T. and Winkler, R. L.: 1983, ‘Limits for the Precision and Value of Information from Dependent Sources’, Unpublished manuscript, University of Indiana, Bloomington, Indiana.Google Scholar
  4. Congressional Research Service: 1976, A Primer on Climatic Variation and Change, U.S. Government Printing Office, Washington.Google Scholar
  5. Council on Environmental Quality: 1980, ‘The Global 2000 Report to the President: Entering the Twenty-First Century’, Washington, D.C. U.S. Government Printing Office.Google Scholar
  6. Daan, H. and Murphy, A. H.: 1982, ‘Subjective Probability Forecasting in The Netherlands: Some Operational and Experimental Results’, Meteorol. Rund. 35, 99–112.Google Scholar
  7. Einhorn, H. J. and Hogarth, R. M.: 1982, ‘Prediction, Diagnosis, and Causal Thinking in Forecasting’, Journal of Forecasting 1, 23–36.Google Scholar
  8. Einhorn, H. J. and Hogarth, R. M.: 1981, ‘Behavioral Decision Theory: Processes of Judgment and Choice’, Annual Review of Psychology 32, 53–88.Google Scholar
  9. Einhorn, H. J., Hogarth, R. M., and Klempner, E.: 1977, ‘Quality of Group Judgment’, Psychological Bulletin 84, 158–172.Google Scholar
  10. Fischer, G. W.: 1981, ‘When Oracles Fail - A Comparison of Four Procedures for Aggregating Subjective Probability Forecasts’, Organizational Behavior and Human Performance 28, 96–110.Google Scholar
  11. Fischhoff, B.: 1982, lsDebiasing’, in D. Kahneman, P. Slovic, and A. Tversky (eds.) Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press, New York.Google Scholar
  12. Fischhoff, B. and MacGregor, D.: 1982, ‘Subjective Confidence in Forecasts’, Journal of Forecasting 1, 155–172.Google Scholar
  13. Glantz, M. H., Robinson, J., and Krenz, M.: 1982, ‘Improving the Science of Climate-Related Impact Studies: A Review of Past Experience’, in William Clark (ed.), Carbon Dioxide Review: 1982, Oxford University Press, New York.Google Scholar
  14. Guilford, J. P.: 1954, Psychometric Methods, McGraw Hill, New York, p. 25.Google Scholar
  15. Hammond, K. R., Anderson, B. F., Sutherland, J., and Marvin, B.: 1982, ‘Improving Scientists’ Judgments of Risk, University of Colorado at Boulder, Center for Research on Judgment and Policy, Report No. 239.Google Scholar
  16. Hammond, K. R., McClelland, G. H., and Mumpower, J.: 1980, Human Judgment and Decision Making: Theories, Methods and Procedures, Praeger, New York.Google Scholar
  17. Hammond, K. R., Rohrbaugh, J., Mumpower, J., and Adelman, L.: 1977, ‘Social Judgment Theory: Applications in Policy Formation’, in M. F. Kaplan and S. Schwartz (eds.), Human Judgment and Decision Processes in Applied Settings, Academic Press, New York.Google Scholar
  18. Hammond, K. R., Stewart, T. R., Brehmer, B., and Steinmann, D. O.: 1975, ‘Social Judgment Theory’, in M. F. Kaplan and S. Schwartz (eds.), Human Judgment and Decision Processes, Academic Press, New York.Google Scholar
  19. Hogarth, R. M.: 1980, Judgment and Choice: The Psychology of Decision, Wiley, Chichester, England.Google Scholar
  20. Hogarth, R. M.: 1977, ‘Methods for Aggregating Opinions’, in H. Jungermann and G. de Zeeuw (eds.), Decision Making and Change in Human Affairs, D. Reidel Publ. Co., Dordrecht, Holland, pp. 231–255.Google Scholar
  21. Hogarth, R. M.: 1975, ‘Cognitive Processes and the Assessment of Subjective Probability Distributions’, J. Amer. Statist. Assoc. 70, 271–289.Google Scholar
  22. Hopkins, R. F. and Puchala, D. J. (eds.): 1978, The Global Political Economy of Food, U. of Wisconsin Press, Madison, Wisconsin, p. 2.Google Scholar
  23. Koriat, A., Lichtenstein, S., and Fischhoff, B.: 1980, ‘Reasons for Confidence’, J. Experimental Psych.: Human Learning and Memory 6, 107–118.Google Scholar
  24. Kraemer, R. S.: 1978, ‘Meeting Reviews: Session on Climatic Futures at the Annual Meeting of the AAAS, 17 February, 1978’, Bulletin of the American Meteorological Society 59(7), 822–23.Google Scholar
  25. Lichtenstein, S., Fischhoff, B., and Phillips, E. D.: 1982, ‘Calibration of Probabilities: The State of the Art to 1980’, in D. Kahneman, P. Slovic, and A. Tversky (eds.), Judgment under Uncertainty: Heuristics and Biases, Cambridge University Press, New York.Google Scholar
  26. Linstone, H. A. and Turoff, M.: 1975, The Delphi Method: Techniques and Applications, Addison-Wesley, Reading, Mass.Google Scholar
  27. McKay, G. A. and Williams, G. D. V.: 1982, ‘Crop Yields and Climate Change to the Year 2000, Volume I (Book Review)’, Bulletin of the American Meteorological Society 63, 427.Google Scholar
  28. Murphy, A. H. and Brown, B. G.: 1984, ‘A Comparative Evaluation of Objective and Subjective Weather Forecasts in the United States’, Journal of Forecasting 3.Google Scholar
  29. Murphy, A. H. and Daan, H.: 1984, ‘Impacts of Feedback and Experience on the Quality of Subjective Probability Forecasts: Comparison of Results from the First and Second Years of the Zierikzee Experiment’, Monthly Weather Rev. 112, 413–423.Google Scholar
  30. Murphy, A. H. and Winkler, R. L.: 1974, ‘Credible Interval Temperature Forecasting: Some Experimental Results’, Monthly Weather Rev. 102, 784–794.Google Scholar
  31. National Academy of Sciences: 1975, Environmental Impact of Stratospheric Flight. National Academy of Sciences, Washington, D.C.Google Scholar
  32. NDU: 1983, The World Grain Economy and Climate Change to the Year 2000: Implications for Policy. National Defense University Press, Washington, D.C.Google Scholar
  33. NDU: 1980, Crop Yields and Climate Change to the Year 2000, National Defense University, Washington, D.C.Google Scholar
  34. NDU: 1978, Climate Change to the Year 2000, National Defense University, Washington, D.C.Google Scholar
  35. Newell, A. and Simon, H. A.: 1972, Human Problem Solving, Prentice-Hall, Inc., Englewood Cliffs, N.J.Google Scholar
  36. Nunnally, J. C: 1978, Psychometric Theory, 2nd ed., McGraw Hill, New York.Google Scholar
  37. Raiffa, H.; 1968, Decision Analysis: Introductory Lectures on Choices under Uncertainty, Addison-Wesley Publishing Co., Reading, Massachusetts.Google Scholar
  38. Rohrbaugh, J.: 1979, ‘Improving the Quality of Group Judgment: Social Judgment Analysis and the Delphi Technique’, Organizational Behavior and Human Performance 24, 73–92.Google Scholar
  39. Sackman, H.: 1975, Delphi Critique, D.C. Heath and Company, Lexington, Mass.Google Scholar
  40. Schuman, H. and Presser, S.: 1981, Question, and Answers in Attitude Surveys. Academic Press, New York.Google Scholar
  41. Science News: 1978, ‘The 25-Year Forecast: Group Prediction’, 113(8), 116.Google Scholar
  42. Seaver, D. A.: 1976, ‘Assessment of Group Preferences and Group Uncertainty for Decision Making’, Technical Report SSRI 76–4, Social Science Research Institute, University of Southern California, Los Angeles, California.Google Scholar
  43. Seaver, D. A., von Winterfeldt, D., and Edwards, W.: 1978, ‘Eliciting Subjective Probability Distributions on Continuous Varibales’, Organizational Behavior and Human Performance 21, 379–391.Google Scholar
  44. Sellers, W. D.: 1979, ‘Climate Change to the Year 2000: A Book Review’, Bulletin of the American Meteorological Society 60(6), 686.Google Scholar
  45. Slovic, P., Fischhoff, B., and Lichtenstein, S.: 1977, ‘Behavioral Decision Theory’, Annual Review of Psychology 28, 1–39.Google Scholar
  46. Slovic, P. and Lichtenstein, S.: 1971, ‘Comparison of Bayesian and Regression Approaches to the Study of Information Processing in Judgment’, Organizational Behavior and Human Performance 6, 649–744.Google Scholar
  47. Spetzler, C. S. and Stael von Holstein, C.-A.S.: 1975, ‘Probability Encoding in Decision Analysis’, Management Sci. 22, 340–358.Google Scholar
  48. Stewart, T. R.: 1984, ‘Judgment and Forecasting: Methodological Implications of Judgment Research’. Paper prepared for the Conference on Forecasting in the Social and Natural Sciences, Boulder, CO, June 10–13.Google Scholar
  49. Tversky, A. and Kahneman, D.: 1981, ‘The Framing of Decisions and the Rationality of Choice’, Science 211, 453–458.Google Scholar
  50. Tversky, A. and Kahneman, D.: 1974, ‘Judgment under Uncertainty: Heuristics and Biases’, Science 185, 1124–1131.Google Scholar
  51. Tversky, A. and Kahneman, D.: 1971, ‘The Belief in the Law of Small Numbers’, Psychological Bulletin 76, 105–10.Google Scholar
  52. U.N. FAO: 1979, ‘Scanning the Future for Climate Change’, CERES 12, 7.Google Scholar
  53. U.S. Central Intelligence Agency: 1974, A Study of Climatological Research as It Pertains to Intelligence Problems, Document Expediting Project, Library of Congress, Washington, D.C.Google Scholar
  54. Wallsten, T.S. and Budescu, D.V.: 1983, ‘Encoding Subjective Probabilities: A Psychological and Psychometric Review’, Management Science 29, 151–173.Google Scholar
  55. Winkler, R.L.: 1968, ‘The Consensus of Subjective Probability Distributions’, Management Science B 15, 61–75.Google Scholar
  56. Winkler, R. L. and Murphy, A.H.: 1979, ‘The Use of Probabilities in Forecasts of Maximum and Minimum Temperatures’, Meteorological Magazine 108, 317–329.Google Scholar

Copyright information

© D. Reidel Publishing Company 1985

Authors and Affiliations

  • Thomas R. Stewart
    • 1
  • Michael H. Glantz
    • 1
  1. 1.Environmental and Societal Impacts Group, National Center for Atmospheric ResearchBoulderUSA

Personalised recommendations