Perspectives on Expertise in the Aggregation of Judgments

  • Gene Rowe

Abstract

Various approaches exist by which the response of a number of experts (or “judges”) may be combined in order to attempt to achieve assessment superior to that which might be attained by merely accepting an individual recommendation. Such approaches have been classified, according to Ferrell (1985), into those of “mathematical,” “behavioral,” and “mixed” type. Briefly, “mathematical” approaches entail the statistical aggregation of a number of judges into a single estimate, while “behavioral” approaches allow the full interaction of group members until some form of consensus is achieved, and “mixed” type involves components of both these approaches.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Armstrong, J. S. (1986). Research on forecasting: A quarter-century review, 1960–1984. Interfaces, 16(11), 89–109.Google Scholar
  2. Ashton, A. H. (1982). An empirical study of budget-related predictions of corporate executives. Journal of Accounting Research, 20(2), 440–449.CrossRefGoogle Scholar
  3. Ashton, A. H., & Ashton, R. H. (1985). Aggregating subjective forecasts: Some empirical results. Management Science, 31, 1499–1508.Google Scholar
  4. Ashton, R. H. (1986). Combining the judgments of experts: How many and which ones? Organizational Behavior and Human Decision Processes, 38, 405–414.CrossRefGoogle Scholar
  5. Best, R. J. (1974). An experiment in Delphi estimation in marketing decision making. Journal of Marketing Research, 11, 448–452.CrossRefGoogle Scholar
  6. Bender, A. D., Strack, A. E., Ebright, G. W., & von Haunalter, G. (1969). Delphi study examines developments in medicine. Futures, 1, 289–303.CrossRefGoogle Scholar
  7. Brockhoff, K. (1975). The performance of forecasting groups in computer dialogue and face to face discussion. In H. Linstone & M. Turoff (Eds.), The Delphi Method: Techniques and applications. London: Addison-Wesley.Google Scholar
  8. Bunn, D. (1989). Forecasting with more than one model. Journal of Forecasting, 8(3), 161–166.CrossRefGoogle Scholar
  9. Clemen, R. T. (1989). Combining forecasts: A review and annotated bibliography. International Journal of Forecasting, 5, 559–583.CrossRefGoogle Scholar
  10. Dalkey, N. C., & Brown, B. (1971). Comparison of group judgment techniques with short-range predictions and almanac questions. The RAND Corporation, R-678-ARPA.Google Scholar
  11. Dalkey, N. C., Brown, B., & Cochran, S. W. (1970). The Delphi Method III: Use of self-ratings to improve group estimates. Technological Forecasting, 1, 283–291.CrossRefGoogle Scholar
  12. Dalkey, N. C., & Helmer, O. (1963). An experimental application of the Delphi Method to the use of experts. Management Science, 9, 458–467.Google Scholar
  13. Dawes, R. (1988). Rational choice in an uncertain world. San Diego: Harcourt Brace, Jovanovich.Google Scholar
  14. Pawes, R. M., & Corrigan, B. (1974). Linear models in decision making. Psychological Bulletin, 81, 95–106.CrossRefGoogle Scholar
  15. DeGroot, M. H. (1974). Reaching a consensus. Journal of the American Statistical Association, 69, 118–121.CrossRefGoogle Scholar
  16. Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1975). Group techniques for program planning. Glenview, 111.: Scott Foresman.Google Scholar
  17. Deutsch, M., & Gerard, H. B. (1955). A study of normative and informational social influences upon individual judgment. Journal of Abnormal and Social Psychology, 51, 629–636.CrossRefGoogle Scholar
  18. Eils, L. C., & John, R. S. (1980). A criterion validation of multiattribute utility analysis and of group communication strategy. Organizational Behavior and Human Performance, 25, 268–288.CrossRefGoogle Scholar
  19. Einhorn, H. J. (1974). Expert judgment: some necessary conditions and an example. Journal of Applied Psychology, 59 (5), 562–571.CrossRefGoogle Scholar
  20. Einhorn, H. J., & Hogarth, R. M. (1975). Unit weighting schemes for decision making. Organizational Behavior and Human Performance, 13, 171–192.CrossRefGoogle Scholar
  21. Einhorn, H. J., Hogarth, R. M., & Klempner, E. (1977). Quality of group judgment. Psychological Bulletin, 84, 158–172.CrossRefGoogle Scholar
  22. Ferrell, W. R. (1985). Combining individual judgments. In G. Wright (Ed.), Behavioral decision making. New York: Plenum.Google Scholar
  23. Fischer, G. W. (1981). When oracles fail—A comparison of four procedures for aggregating subjective probability forecasts. Organizational Behavior and Human Performance, 28, 96–110.CrossRefGoogle Scholar
  24. Flores, B. E., & White, E. M. (1989). Subjective vs objective combining of forecasts: An experiment. Journal of Forecasting, 8, 331–341.CrossRefGoogle Scholar
  25. Fraser, C., & Foster, D. (1984). Social groups, nonsense groups and group polarization. In H. Tajfel (Ed), Group Processes, New York: Academic Press.Google Scholar
  26. Granger, C. W. J., & Newbold, P. (1975). Economic forecasting: The atheist’s viewpoint. In Renton, G. A. (Ed.), Modelling the economy. London: Heineman.Google Scholar
  27. Gustafson, D. H., Shukla, R. K., Delbecq, A., & Walster, G. W. (1973). A comparison study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups and nominal groups. Organizational Behavior and Human Performance, 9, 280–291.CrossRefGoogle Scholar
  28. Hackman, J. R., & Morris, C. G. (1975). Group tasks, group interaction process and group performance effectiveness: A review and proposed integration. Advances in Experimental Social Psychology, 8, 45–99.CrossRefGoogle Scholar
  29. Hall, J., & Watson, W. H. (1971). The effects of a normative intervention on group decision-making performance. Human Relations, 23, 299–317.CrossRefGoogle Scholar
  30. Hastie, R. (1986). Experimental evidence on group accuracy. In B. Grafman & G. Owen (Eds.), Decision research (Vol. 2), Greenwich, CT: JAI Press.Google Scholar
  31. Hill, G. W. (1982). Group versus individual performance: Are N+l heads better than one? Psychological Bulletin, 91(3), 517–539.CrossRefGoogle Scholar
  32. Hill, K. Q., & Fowles, J. (1975). The methodological worth of the Delphi forecasting technique. Technological Forecasting and Social Change, 7, 179–192.CrossRefGoogle Scholar
  33. Hoffman, L. R. (1965). Group problem solving. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 2). New York: Academic Press.Google Scholar
  34. Hogarth, R. M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21, 40–46.CrossRefGoogle Scholar
  35. Hogarth, R. M., & Makridakis, S. (1981). Forecasting and planning: An evaluation. Management Science, 27, 115–138.CrossRefGoogle Scholar
  36. Janis, I. (1972). Victims of groupthink, Boston: Houghton Mifflin.Google Scholar
  37. Jenkins, G. M. (1974). Discussion of a paper by Newbold and Granger. Journal of the Royal Statistical Society A, 137, 148–150.Google Scholar
  38. Jolson, M. A., & Rossow, G. (1971). The Delphi process in marketing decision making. Journal of Marketing Research, 8, 443–448.CrossRefGoogle Scholar
  39. Kleinmuntz, B. (1989). Why we still use our heads instead of formulas: Towards an integrative approach. Psychological Bulletin, in press.Google Scholar
  40. Larreche, J. C., & Moinpour, R. (1983). Managerial judgment in marketing: The concept of expertise. Journal of Marketing Research, 20, 110–121.CrossRefGoogle Scholar
  41. Libby, R., & Blashfield, R. K. (1978). Performance of a compolite as a function of the number of judges. Organizational Behavior and Human Performance, 21, 121–129.CrossRefGoogle Scholar
  42. Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980. In Kahneman, D., Slovic, P., & Tversky, A. (Eds.), Judgment under uncertainty: Heuristics and biases, New York: Cambridge University Press.Google Scholar
  43. Linstone, H. A. (1978). The Delphi Technique. In R. B. Fowles (Ed.), Handbook of futures research, Westport, CT: Greenwood Press.Google Scholar
  44. Linstone, H., & Turoff, M. (1975). The Delphi Method: Techniques and applications, London: Addison-Wesley.Google Scholar
  45. Lock, A. (1987). Integrating group judgments in subjective forecasts. In G. Wright & P. Ayton (Eds.), Judgmental forecasting, Chichester: Wiley.Google Scholar
  46. Lorge, I., Fox, D., Davitz, J., & Brenner, M. (1958). A survey of studies contrasting the quality of group performance and individual performance. Psychological Bulletin, 55, 337–372.PubMedCrossRefGoogle Scholar
  47. McKinnon, W. J. (1966). Development of the SPAN technique for making decisions in human groups. American Behavioral Scientist, 9, 9–13.CrossRefGoogle Scholar
  48. Martino, J. (1983). Technological forecasting for decision-makers (2nd ed.). New York: Elsevier.Google Scholar
  49. Nisbett, R., & Ross, R. L. (1980). Human inference: Strategies and shortcomings of social judgment. Englewood Cliffs, N.J.: Prentice-Hall.Google Scholar
  50. Northcroft, M. A., & Neale, G. B. (1987). Experts, amateurs and real-estate: An anchoring and adjust perspective in property pricing decisions. Organizational Behavior and Human Decision Processes, 39, 84–97.CrossRefGoogle Scholar
  51. Parente, F. J., & Anderson-Parente, J. K. (1987). Delphi inquiry systems. In G. Wright & P. Ayton (Eds.), Judgmental forecasting, Chichester: Wiley.Google Scholar
  52. Riggs, W. E. (1983). The Delphi Method: An experimental evaluation. Technological Forecasting and Social Change, 23, 89–94.CrossRefGoogle Scholar
  53. Rohrbaugh, J. (1979). Improving the quality of group judgment: Social judgment analysis and the Delphi technique. Organizational Behavior and Human Performance, 24, 73–92.CrossRefGoogle Scholar
  54. Rowse, G. L., Gustafson, D. H., & Ludke, R. L. (1974). Comparison of rules of aggregating subjective likelihood ratios. Organizational Behavior and Human Performance, 12, 274–285.CrossRefGoogle Scholar
  55. Salancik, J. R., Wenger, W., & Heifer, E. (1971). The construction of delphi event statements. Technological Forecasting and Social Change, 3, 65–73.CrossRefGoogle Scholar
  56. Seaver, D. A. (1979). Assessing probability with multiple individuals, Unpublished doctoral dissertation, University of Southern California, Los Angeles.Google Scholar
  57. Sniezek, J. A., & Henry, R. A. (1989). Accuracy and confidence in group judgment. Organizational Behavior and Human Decision Processes, 43, 1–28.CrossRefGoogle Scholar
  58. Steiner, I. D. (1972). Group process and productivity. New York: Academic Press.Google Scholar
  59. Tuckman, J., & Lorge, I. (1962). Individual ability as a determinant of group superiority. Human Relations, 15, 45–51.CrossRefGoogle Scholar
  60. Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124–1131.CrossRefPubMedGoogle Scholar
  61. Uecker, W. L. (1982). The quality of group performance in simplified information evaluation. Journal of Accounting Research, 20, 388–402.CrossRefGoogle Scholar
  62. Vande Ven, A. H., & Delbecq, A. L. (1971). Nominal versus interacting group processes for committee decision making effectiveness. Academic Management Journal, 14, 203–213.CrossRefGoogle Scholar
  63. Winkler, R. L. (1971). Probablistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685.CrossRefGoogle Scholar

Copyright information

© Plenum Press, New York 1992

Authors and Affiliations

  • Gene Rowe
    • 1
  1. 1.Bristol Business SchoolFrenchay, BristolEngland

Personalised recommendations