Theory and Decision

, Volume 73, Issue 4, pp 671–685 | Cite as

The Group Calibration Index: a group-based approach for assessing forecasters’ expertise when external outcome data are missing

Article
  • 86 Downloads

Abstract

The Group Calibration Index (GCI) provides a means of assessing the quality of forecasters’ predictions in situations that lack external feedback or outcome data. The GCI replaces the missing outcome data with aggregated ratings of a well-defined reference group. A simulation study and two experiments show how the GCI classifies forecaster performance and distinguishes between forecasters with restricted information and those with complete information. The results also show that under certain circumstances, where members of the reference group have high-quality information, the new GCI will outperform expert classification that is based on traditional calibration indices.

Keywords

Group decisions Experts Calibration Uncertainty 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Brier G. W. (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Review 75: 1–3CrossRefGoogle Scholar
  2. Dawes R. M. (1994) House of cards: Psychology and psychotherapy built of myth. Free Press, New YorkGoogle Scholar
  3. Fischer I., Budescu D. V. (2005) When “do those who know more also know more about how much they know”: The development of confidence and performance in decision-making. Organizational Behavior and Human Decision Processes 98(1): 39–53CrossRefGoogle Scholar
  4. Gustafson D. H., Shukla R. K., Delbecq A., Walster G. W. (1973) A comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups. Organizational Behavior and Human Performance 9: 280–291CrossRefGoogle Scholar
  5. Harvey N., Fischer I. (2005) Development of experience-based judgment and decision making: The role of Outcome Feedback. In: Betsch T., Haberstroh S. (eds) The routines of decision making. Erlbaum, Mahwah, NJ, pp 119–137Google Scholar
  6. Keren G. (1987) Facing uncertainty in the game of bridge: A calibration study. Organizational Behavior and Human Decision Processes 39: 98–114CrossRefGoogle Scholar
  7. Larrick R. P., Soll B. J. (2006) Intuitions about combining opinions: Misappreciation of the averaging principle. Management Science 52(1): 111–127CrossRefGoogle Scholar
  8. Lichtenstein S., Fischhoff B. (1977) Do those who know more also know more about they know?. Organizational Behavior and Human Performance 20: 159–183CrossRefGoogle Scholar
  9. Murphy A. H. (1973) A new vector partition of the probability score. Journal of Applied Meteorology 12: 595–600CrossRefGoogle Scholar
  10. Oskamp, S. (1965). Overconfidence in case-study judgments. The Journal of Consulting Psychology, 29, 261–265. Reprinted in: Kahneman, D., Slovic, P., Tversky, A. (1982). Judgment under Uncertainty, 287–293. (Cambridge: Cambridge University Press).Google Scholar
  11. Prelec D. (2004) A Baysian truth serum for subjective data. Science 306: 462–466CrossRefGoogle Scholar
  12. Rowe G., Wright G. (2001) Expert opinions in Forecasting: The Role of the Delphi Technique. In: Armstrong J. S. (eds) Principles of forecasting—a handbook for researchers and practitioners. Kluwer Academic Publishers, Boston, pp 125–144Google Scholar
  13. Shuford E. H. (1961) Absolute judgments of discrete quantities randomly distributed over time. Journal of Experimental Psychology 61: 430–436CrossRefGoogle Scholar
  14. Sterman J. (1994) Learning in and about complex systems. System dynamics Review 16: 291–330CrossRefGoogle Scholar
  15. Stevens S. S. (1975) Psychophysics: Introduction to its perceptual, neural, and social prospects. John Wiley & Sons, New York, pp 13–36Google Scholar
  16. Van de Ven A. H., Delbecq A. L. (1974) The effectiveness of nominal, delphi, and interacting group decision making processes. The Academy of Management Journal 17: 605–621CrossRefGoogle Scholar
  17. Vincenz, S. (1955). Bałaguły, On the High Uplands, translated by H. C. Stevens (pp. 248–267). New York: RoyGoogle Scholar
  18. Wagenaar W. A., Keren G. B. (1985) Calibration of probability assessment by professional Blackjack dealers, statistical experts, and lay people. Organizational Behavior and Human Decision Processes 36: 406–416CrossRefGoogle Scholar
  19. Wallsten T. S., Budescu D. V. (1983) Encoding subjective probabilities: A psychological and psychometric review. Management Science 29: 151–173CrossRefGoogle Scholar
  20. Yates J. F. (1982) External correspondence: Decompositions of the mean probability score. Organizational-Behavior-and-Human-Decision-Processes 30: 132–156CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC. 2011

Authors and Affiliations

  1. 1.University of HaifaHaifaIsrael
  2. 2.The Open University of IsraelRaananaIsrael

Personalised recommendations