Expert Opinions in Forecasting: The Role of the Delphi Technique

  • Gene Rowe
  • George Wright
Part of the International Series in Operations Research & Management Science book series (ISOR, volume 30)

Abstract

Expert opinion is often necessary in forecasting tasks because of a lack of appropriate or available information for using statistical procedures. But how does one get the best forecast from experts? One solution is to use a structured group technique, such as Delphi, for eliciting and combining expert judgments. In using the Delphi technique, one controls the exchange of information between anonymous panelists over a number of rounds (iterations), taking the average of the estimates on the final round as the group judgment. A number of principles are developed here to indicate how to conduct structured groups to obtain good expert judgments. These principles, applied to the conduct of Delphi groups, indicate how many and what type of experts to use (five to 20 experts with disparate domain knowledge); how many rounds to use (generally two or three); what type of feedback to employ (average estimates plus justifications from each expert); how to summarize the final forecast (weight all experts’ estimates equally); how to word questions (in a balanced way with succinct definitions free of emotive terms and irrelevant information); and what response modes to use (frequencies rather than probabilities or odds, with coherence checks when feasible). Delphi groups are substantially more accurate than individual experts and traditional groups and somewhat more accurate than statistical groups (which are made up of noninteracting individuals whose judgments are aggregated). Studies support the advantage of Delphi groups over traditional groups by five to one with one tie, and their advantage over statistical groups by 12 to two with two ties. We anticipate that by following these principles, forecasters may be able to use structured groups to harness effectively expert opinion.

Keywords

Delphi expertise interacting groups statistical groups 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Arkes, H. (2001), “Overconfidence in judgmental forecasting,” in J..S. Armstrong (ed.), Principles of Forecasting. Norwell, MA.: Kluwer Academic Publishers.Google Scholar
  2. Armstrong, J. S. (1985), Long Range Forecasting: From Crystal Ball to Computer,2nd ed., New York: Wiley. (Full text at http://hops.wharton.upenn.edu/forecast.)Google Scholar
  3. Bardecki, M.J. (1984), “Participants’ response to the Delphi method: An attitudinal perspective,” Technological Forecasting and Social Change, 25, 281–292.CrossRefGoogle Scholar
  4. Beach, L. R. and L. D. Phillips (1967), “Subjective probabilities inferred from estimates and bets,” Journal of Experimental Psychology, 75, 354–259.CrossRefGoogle Scholar
  5. Best, R. J. (1974), “An experiment in Delphi estimation in marketing decision making,” Journal of Marketing Research, 11, 448–452.CrossRefGoogle Scholar
  6. Boje, D. M. and J. K. Murnighan (1982), “Group confidence pressures in iterative decisions,” Management Science, 28, 1187–1196.CrossRefGoogle Scholar
  7. Brockhoff, K. (1975), “The performance of forecasting groups in computer dialogue and face to face discussions,” in H. Linstone and M. Turoff (eds.), The Delphi Method: Techniques and Applications. London: Addison-Wesley.Google Scholar
  8. Cooper, A., C. Woo and W. Dunkelberger (1988), “Entrepreneurs perceived chances of success,” Journal of Business Venturing, 3, 97–108.CrossRefGoogle Scholar
  9. Dalkey, N.C., B. Brown and S. W. Cochran (1970), “The Delphi Method III: Use of self- ratings to improve group estimates,” Technological Forecasting, 1, 283–291.CrossRefGoogle Scholar
  10. Dawes, R. M. (1982), “The robust beauty of improper linear models in decision making,” in D. Kahneman, P. Slovic and A. Tversky (eds.), Judgement Under Uncertainty: Heuristics and Biases. Cambridge: Cambridge University Press.Google Scholar
  11. Dietz, T. (1987), “Methods for analyzing data from Delphi panels: Some evidence from a forecasting study,” Technological Forecasting and Social Change, 31, 79–85.CrossRefGoogle Scholar
  12. Erffineyer, R. C., E. S. Erffineyer and I. M. Lane (1986), “The Delphi technique: An empirical evaluation of the optimal number of rounds,” Group and Organization Studies, 11, 120–128.CrossRefGoogle Scholar
  13. Erffineyer, R. C. and I.M. Lane (1984), “Quality and acceptance of an evaluative task: The effects of four group decision-making formats,” Group and Organization Studies, 9, 509–529.CrossRefGoogle Scholar
  14. Fischer, G. W. (1981), “When oracles fail-a comparison of four procedures for aggregating subjective probability forecasts,” Organizational Behavior and Human Performance, 28, 96–110.CrossRefGoogle Scholar
  15. Gigerenzer, G. (1994), “Why the distinction between single event probabilities and frequencies is important for psychology (and vice-versa),” in G. Wright and P. Ayton (eds.), Subjective Probability. Chichester, U.K.: Wiley.Google Scholar
  16. Goodwin, P. and G. Wright (1998), Decision Analysis for Management Judgment, 2nd ed. Chichester, U.K.: Wiley.Google Scholar
  17. Gustafson, D. H., R. K. Shukla, A. Delbecq and G. W. Walster (1973), “A comparison study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups and nominal groups,” Organizational Behavior and Human Performance, 9, 280–291.CrossRefGoogle Scholar
  18. Hauser, P.M. (1975), Social Statistics in Use. New York: Russell Sage.Google Scholar
  19. Hill, G. W. (1982), “Group versus individual performance: Are N+1 heads better than one?” Psychological Bulletin, 91, 517–539.CrossRefGoogle Scholar
  20. Hill, K. Q. and J. Fowles (1975), “The methodological worth of the Delphi forecasting technique,” Technological Forecasting and Social Change, 7, 179–192.CrossRefGoogle Scholar
  21. Hogarth, R. M. (1978), “A note on aggregating opinions,” Organizational Behavior and Human Performance, 21, 40–46.CrossRefGoogle Scholar
  22. Jolson, M. A. and G. Rossow (1971), “The Delphi process in marketing decision making,” Journal of Marketing Research, 8, 443–448.CrossRefGoogle Scholar
  23. Kahneman, D. and D. Lovallo (1993), “Timid choices and bold forecasts: A cognitive perspective on risk taking,” Management Science, 39, 17–31.CrossRefGoogle Scholar
  24. Larreché, J. C. and R. Moinpour (1983), “Managerial judgment in marketing: The concept of expertise,” Journal of Marketing Research, 20, 110–121.CrossRefGoogle Scholar
  25. Linstone, H. A. (1975), “Eight basic pitfalls: A checklist,” in H. Linstone and M. Turoff (eds.), The Delphi Method: Techniques and Applications. London: Addision-Wesley.Google Scholar
  26. Linstone, H. A. and M. Turoff (1975), The Delphi Method: Techniques and Applications. London: Addision-Wesley.Google Scholar
  27. Lock, A. (1987), “Integrating group judgments in subjective forecasts,” in G. Wright and P. Ayton (eds.), Judgmental Forecasting. Chichester, U.K.: Wiley.Google Scholar
  28. MacGregor, D. G. (2001), “Decomposition for judgmental forecasting and estimation,” in J. S. Armstrong (ed.), Principles of Forecasting. Norwell, MA.: Kluwer Academic Publishers.Google Scholar
  29. Martino, J. (1983), Technological Forecasting for Decision Making, ( 2nd ed. ). New York: American Elsevier.Google Scholar
  30. McClelland, A. G. R. and F. Bolger (1994), “The calibration of subjective probabilities: Theories and models 1980–1994,” in G. Wright and P. Ayton (eds.), Subjective Probability. Chichester, U.K.: Wiley.Google Scholar
  31. Meehl, P. E. (1954), Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence. Minneapolis: University of Minnesota Press.CrossRefGoogle Scholar
  32. Miner, F. C. (1979), “A comparative analysis of three diverse group decision making approaches,” Academy of Management Journal, 22, 81–93.CrossRefGoogle Scholar
  33. Noelle-Neuman, E. (1970), “Wanted: Rules for wording structured questionnaires,” Public Opinion Quarterly, 34, 190–201.Google Scholar
  34. Parenté, F.J., J. K. Anderson, P. Myers and T. O’Brien (1984), “An examination of factors contributing to Delphi accuracy,” Journal of Forecasting, 3, 173–182.CrossRefGoogle Scholar
  35. Parenté, F. J. and J. K. Anderson-Parenté (1987), “Delphi inquiry systems,” in G. Wright and P. Ayton (eds.), Judgmental Forecasting. Chichester, U.K.: Wiley.Google Scholar
  36. Payne, S.L. (1951), The Art of Asking Questions. Princeton, NJ: Princeton University Press.Google Scholar
  37. Riggs, W. E. (1983), “The Delphi method: An experimental evaluation,” Technological Forecasting and Social Change, 23, 89–94.CrossRefGoogle Scholar
  38. Rohrbaugh, J. (1979), “Improving the quality of group judgment: Social judgment analysis and the Delphi technique,” Organizational Behavior and Human Performance, 24, 73–92.CrossRefGoogle Scholar
  39. Rowe, G. and G. Wright (1996), “The impact of task characteristics on the performance of structured group forecasting techniques,” International Journal of Forecasting, 12, 73–89.CrossRefGoogle Scholar
  40. Rowe, G. and G. Wright (1999), “The Delphi technique as a forecasting tool: Issues and analysis,” International Journal of Forecasting, 15, 353–375. (Commentary follows on pp. 377–381.)Google Scholar
  41. Rowe, G., G. Wright and F. Bolger (1991), “The Delphi technique: A reevaluation of research and theory,” Technological Forecasting and Social Change, 39, 235–251.CrossRefGoogle Scholar
  42. Sackman, H. (1975), Delphi Critique. Lexington, MA: Lexington Books.Google Scholar
  43. Salancik, J. R., W. Wenger and E. Helfer (1971), “The construction of Delphi event statements,” Technological Forecasting and Social Change, 3, 65–73.CrossRefGoogle Scholar
  44. Scheibe, M., M. Skutsch and J. Schofer (1975), “Experiments in Delphi methodology,” in H. Linstone and M. Turoff (eds.), The Delphi Method: Techniques and Applications. London: Addison-Wesley.Google Scholar
  45. Sniezek, J. A. (1989), “An examination of group process in judgmental forecasting,” International Journal of Forecasting, 5, 171–178.CrossRefGoogle Scholar
  46. Sniezek, J. A. (1990), “A comparison of techniques for judgmental forecasting by groups with common information,” Group and Organization Studies, 15, 5–19.CrossRefGoogle Scholar
  47. Sniezek, J. A. and T. Buckley (1991), “Confidence depends on level of aggregation,” Journal of Behavioral Decision Making, 4, 263–272.CrossRefGoogle Scholar
  48. Stewart, T.R. (1987), “The Delphi technique and judgmental forecasting,” Climatic Change, 11, 97–113.CrossRefGoogle Scholar
  49. Stewart, T.R. (2001), “Improving reliability in judgmental forecasts,” in J. S. Armstrong (ed.), Principles of Forecasting. Norwell, MA.: Kluwer Academic Publishers.Google Scholar
  50. Sudman, S. and N. Bradburn (1982), Asking Questions. San Francisco: Josey-Bass.Google Scholar
  51. Tversky, A. and D. Kahneman (1974), “Judgment under uncertainty: Heuristics and biases,” Science, 185, 1124–1131.CrossRefGoogle Scholar
  52. Tversky, A. and D. Kahneman (1981), “The framing of decisions and the psychology of choice,” Science, 211, 453–458.CrossRefGoogle Scholar
  53. Van de Ven, A. H. and A. L. Delbecq (1971), “Nominal versus interacting group processes for committee decision making effectiveness,” Academic Management Journal, 14, 203–213.CrossRefGoogle Scholar
  54. Van de Ven, A. H. and A. L. Delbecq (1974), “The effectiveness of nominal, Delphi, and interacting group decision making processes,” Academy of Management Journal, 17, 605–621.CrossRefGoogle Scholar
  55. Welty, G. (1974), “The necessity, sufficiency and desirability of experts as value forecasters,” in W. Leinfellner and E. Kohler (eds.), Developments in the Methodology ofSocial Science. Boston: Reidel.Google Scholar
  56. Wright, G. and P. Ayton (1994), Subjective Probability. Chichester, U.K.: Wiley.Google Scholar
  57. Wright, G., G. Rowe, F. Bolger and J. Gammack (1994), “Coherence, calibration and expertise in judgmental probability forecasting,” Organizational Behavior and Human Decision Processes, 57, 1–25.CrossRefGoogle Scholar
  58. Wright, G., C. Saunders and P. Ayton (1988), “The consistency, coherence and calibration of holistic, decomposed and recomposed judgmental probability forecasts,” Journal of Forecasting, 7, 185–199.CrossRefGoogle Scholar
  59. Wright, G. and P. Whalley (1983), “The supra-additivity of subjective probability,” in B. Stigum and F. Wenstop (eds.), Foundations of Risk and Utility Theory with Applications. Dordrecht: Reidel.Google Scholar

Copyright information

© Springer Science+Business Media New York 2001

Authors and Affiliations

  • Gene Rowe
    • 1
  • George Wright
    • 2
  1. 1.Institute of Food ResearchNorwich Research ParkUK
  2. 2.Strathclyde Graduate Business SchoolStrathclyde UniversityUK

Personalised recommendations