Abstract
Expert opinion is often necessary in forecasting tasks because of a lack of appropriate or available information for using statistical procedures. But how does one get the best forecast from experts? One solution is to use a structured group technique, such as Delphi, for eliciting and combining expert judgments. In using the Delphi technique, one controls the exchange of information between anonymous panelists over a number of rounds (iterations), taking the average of the estimates on the final round as the group judgment. A number of principles are developed here to indicate how to conduct structured groups to obtain good expert judgments. These principles, applied to the conduct of Delphi groups, indicate how many and what type of experts to use (five to 20 experts with disparate domain knowledge); how many rounds to use (generally two or three); what type of feedback to employ (average estimates plus justifications from each expert); how to summarize the final forecast (weight all experts’ estimates equally); how to word questions (in a balanced way with succinct definitions free of emotive terms and irrelevant information); and what response modes to use (frequencies rather than probabilities or odds, with coherence checks when feasible). Delphi groups are substantially more accurate than individual experts and traditional groups and somewhat more accurate than statistical groups (which are made up of noninteracting individuals whose judgments are aggregated). Studies support the advantage of Delphi groups over traditional groups by five to one with one tie, and their advantage over statistical groups by 12 to two with two ties. We anticipate that by following these principles, forecasters may be able to use structured groups to harness effectively expert opinion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Arkes, H. (2001), “Overconfidence in judgmental forecasting,” in J..S. Armstrong (ed.), Principles of Forecasting. Norwell, MA.: Kluwer Academic Publishers.
Armstrong, J. S. (1985), Long Range Forecasting: From Crystal Ball to Computer,2nd ed., New York: Wiley. (Full text at http://hops.wharton.upenn.edu/forecast.)
Bardecki, M.J. (1984), “Participants’ response to the Delphi method: An attitudinal perspective,” Technological Forecasting and Social Change, 25, 281–292.
Beach, L. R. and L. D. Phillips (1967), “Subjective probabilities inferred from estimates and bets,” Journal of Experimental Psychology, 75, 354–259.
Best, R. J. (1974), “An experiment in Delphi estimation in marketing decision making,” Journal of Marketing Research, 11, 448–452.
Boje, D. M. and J. K. Murnighan (1982), “Group confidence pressures in iterative decisions,” Management Science, 28, 1187–1196.
Brockhoff, K. (1975), “The performance of forecasting groups in computer dialogue and face to face discussions,” in H. Linstone and M. Turoff (eds.), The Delphi Method: Techniques and Applications. London: Addison-Wesley.
Cooper, A., C. Woo and W. Dunkelberger (1988), “Entrepreneurs perceived chances of success,” Journal of Business Venturing, 3, 97–108.
Dalkey, N.C., B. Brown and S. W. Cochran (1970), “The Delphi Method III: Use of self- ratings to improve group estimates,” Technological Forecasting, 1, 283–291.
Dawes, R. M. (1982), “The robust beauty of improper linear models in decision making,” in D. Kahneman, P. Slovic and A. Tversky (eds.), Judgement Under Uncertainty: Heuristics and Biases. Cambridge: Cambridge University Press.
Dietz, T. (1987), “Methods for analyzing data from Delphi panels: Some evidence from a forecasting study,” Technological Forecasting and Social Change, 31, 79–85.
Erffineyer, R. C., E. S. Erffineyer and I. M. Lane (1986), “The Delphi technique: An empirical evaluation of the optimal number of rounds,” Group and Organization Studies, 11, 120–128.
Erffineyer, R. C. and I.M. Lane (1984), “Quality and acceptance of an evaluative task: The effects of four group decision-making formats,” Group and Organization Studies, 9, 509–529.
Fischer, G. W. (1981), “When oracles fail-a comparison of four procedures for aggregating subjective probability forecasts,” Organizational Behavior and Human Performance, 28, 96–110.
Gigerenzer, G. (1994), “Why the distinction between single event probabilities and frequencies is important for psychology (and vice-versa),” in G. Wright and P. Ayton (eds.), Subjective Probability. Chichester, U.K.: Wiley.
Goodwin, P. and G. Wright (1998), Decision Analysis for Management Judgment, 2nd ed. Chichester, U.K.: Wiley.
Gustafson, D. H., R. K. Shukla, A. Delbecq and G. W. Walster (1973), “A comparison study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups and nominal groups,” Organizational Behavior and Human Performance, 9, 280–291.
Hauser, P.M. (1975), Social Statistics in Use. New York: Russell Sage.
Hill, G. W. (1982), “Group versus individual performance: Are N+1 heads better than one?” Psychological Bulletin, 91, 517–539.
Hill, K. Q. and J. Fowles (1975), “The methodological worth of the Delphi forecasting technique,” Technological Forecasting and Social Change, 7, 179–192.
Hogarth, R. M. (1978), “A note on aggregating opinions,” Organizational Behavior and Human Performance, 21, 40–46.
Jolson, M. A. and G. Rossow (1971), “The Delphi process in marketing decision making,” Journal of Marketing Research, 8, 443–448.
Kahneman, D. and D. Lovallo (1993), “Timid choices and bold forecasts: A cognitive perspective on risk taking,” Management Science, 39, 17–31.
Larreché, J. C. and R. Moinpour (1983), “Managerial judgment in marketing: The concept of expertise,” Journal of Marketing Research, 20, 110–121.
Linstone, H. A. (1975), “Eight basic pitfalls: A checklist,” in H. Linstone and M. Turoff (eds.), The Delphi Method: Techniques and Applications. London: Addision-Wesley.
Linstone, H. A. and M. Turoff (1975), The Delphi Method: Techniques and Applications. London: Addision-Wesley.
Lock, A. (1987), “Integrating group judgments in subjective forecasts,” in G. Wright and P. Ayton (eds.), Judgmental Forecasting. Chichester, U.K.: Wiley.
MacGregor, D. G. (2001), “Decomposition for judgmental forecasting and estimation,” in J. S. Armstrong (ed.), Principles of Forecasting. Norwell, MA.: Kluwer Academic Publishers.
Martino, J. (1983), Technological Forecasting for Decision Making, ( 2nd ed. ). New York: American Elsevier.
McClelland, A. G. R. and F. Bolger (1994), “The calibration of subjective probabilities: Theories and models 1980–1994,” in G. Wright and P. Ayton (eds.), Subjective Probability. Chichester, U.K.: Wiley.
Meehl, P. E. (1954), Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence. Minneapolis: University of Minnesota Press.
Miner, F. C. (1979), “A comparative analysis of three diverse group decision making approaches,” Academy of Management Journal, 22, 81–93.
Noelle-Neuman, E. (1970), “Wanted: Rules for wording structured questionnaires,” Public Opinion Quarterly, 34, 190–201.
Parenté, F.J., J. K. Anderson, P. Myers and T. O’Brien (1984), “An examination of factors contributing to Delphi accuracy,” Journal of Forecasting, 3, 173–182.
Parenté, F. J. and J. K. Anderson-Parenté (1987), “Delphi inquiry systems,” in G. Wright and P. Ayton (eds.), Judgmental Forecasting. Chichester, U.K.: Wiley.
Payne, S.L. (1951), The Art of Asking Questions. Princeton, NJ: Princeton University Press.
Riggs, W. E. (1983), “The Delphi method: An experimental evaluation,” Technological Forecasting and Social Change, 23, 89–94.
Rohrbaugh, J. (1979), “Improving the quality of group judgment: Social judgment analysis and the Delphi technique,” Organizational Behavior and Human Performance, 24, 73–92.
Rowe, G. and G. Wright (1996), “The impact of task characteristics on the performance of structured group forecasting techniques,” International Journal of Forecasting, 12, 73–89.
Rowe, G. and G. Wright (1999), “The Delphi technique as a forecasting tool: Issues and analysis,” International Journal of Forecasting, 15, 353–375. (Commentary follows on pp. 377–381.)
Rowe, G., G. Wright and F. Bolger (1991), “The Delphi technique: A reevaluation of research and theory,” Technological Forecasting and Social Change, 39, 235–251.
Sackman, H. (1975), Delphi Critique. Lexington, MA: Lexington Books.
Salancik, J. R., W. Wenger and E. Helfer (1971), “The construction of Delphi event statements,” Technological Forecasting and Social Change, 3, 65–73.
Scheibe, M., M. Skutsch and J. Schofer (1975), “Experiments in Delphi methodology,” in H. Linstone and M. Turoff (eds.), The Delphi Method: Techniques and Applications. London: Addison-Wesley.
Sniezek, J. A. (1989), “An examination of group process in judgmental forecasting,” International Journal of Forecasting, 5, 171–178.
Sniezek, J. A. (1990), “A comparison of techniques for judgmental forecasting by groups with common information,” Group and Organization Studies, 15, 5–19.
Sniezek, J. A. and T. Buckley (1991), “Confidence depends on level of aggregation,” Journal of Behavioral Decision Making, 4, 263–272.
Stewart, T.R. (1987), “The Delphi technique and judgmental forecasting,” Climatic Change, 11, 97–113.
Stewart, T.R. (2001), “Improving reliability in judgmental forecasts,” in J. S. Armstrong (ed.), Principles of Forecasting. Norwell, MA.: Kluwer Academic Publishers.
Sudman, S. and N. Bradburn (1982), Asking Questions. San Francisco: Josey-Bass.
Tversky, A. and D. Kahneman (1974), “Judgment under uncertainty: Heuristics and biases,” Science, 185, 1124–1131.
Tversky, A. and D. Kahneman (1981), “The framing of decisions and the psychology of choice,” Science, 211, 453–458.
Van de Ven, A. H. and A. L. Delbecq (1971), “Nominal versus interacting group processes for committee decision making effectiveness,” Academic Management Journal, 14, 203–213.
Van de Ven, A. H. and A. L. Delbecq (1974), “The effectiveness of nominal, Delphi, and interacting group decision making processes,” Academy of Management Journal, 17, 605–621.
Welty, G. (1974), “The necessity, sufficiency and desirability of experts as value forecasters,” in W. Leinfellner and E. Kohler (eds.), Developments in the Methodology ofSocial Science. Boston: Reidel.
Wright, G. and P. Ayton (1994), Subjective Probability. Chichester, U.K.: Wiley.
Wright, G., G. Rowe, F. Bolger and J. Gammack (1994), “Coherence, calibration and expertise in judgmental probability forecasting,” Organizational Behavior and Human Decision Processes, 57, 1–25.
Wright, G., C. Saunders and P. Ayton (1988), “The consistency, coherence and calibration of holistic, decomposed and recomposed judgmental probability forecasts,” Journal of Forecasting, 7, 185–199.
Wright, G. and P. Whalley (1983), “The supra-additivity of subjective probability,” in B. Stigum and F. Wenstop (eds.), Foundations of Risk and Utility Theory with Applications. Dordrecht: Reidel.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2001 Springer Science+Business Media New York
About this chapter
Cite this chapter
Rowe, G., Wright, G. (2001). Expert Opinions in Forecasting: The Role of the Delphi Technique. In: Armstrong, J.S. (eds) Principles of Forecasting. International Series in Operations Research & Management Science, vol 30. Springer, Boston, MA. https://doi.org/10.1007/978-0-306-47630-3_7
Download citation
DOI: https://doi.org/10.1007/978-0-306-47630-3_7
Publisher Name: Springer, Boston, MA
Print ISBN: 978-0-7923-7401-5
Online ISBN: 978-0-306-47630-3
eBook Packages: Springer Book Archive