Psychometrika

, Volume 74, Issue 3, pp 531–553 | Cite as

Why Are Experts Correlated? Decomposing Correlations Between Judges

Theory and Methods

Abstract

We derive an analytic model of the inter-judge correlation as a function of five underlying parameters. Inter-cue correlation and the number of cues capture our assumptions about the environment, while differentiations between cues, the weights attached to the cues, and (un)reliability describe assumptions about the judges. We study the relative importance of, and interrelations between these five factors with respect to inter-judge correlation. Results highlight the centrality of the inter-cue correlation. We test the model’s predictions with empirical data and illustrate its relevance. For example, we show that, typically, additional judges increase efficacy at a greater rate than additional cues.

Keywords

information aggregation correlation dependence expert advice 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ariely, D., Au, W.T., Bender, R.H., Budescu, D.V., Dietz, C.B., Gu, H., Wallsten, T.S., & Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6, 130–147. PubMedCrossRefGoogle Scholar
  2. Ashton, R.H. (1986). Combining the judgments of experts: How many and which ones? Organizational Behavior and Human Decision Processes, 38, 405–414. CrossRefGoogle Scholar
  3. Ashton, A.H., & Ashton, R.H. (1985). Aggregating subjective forecasts: Some empirical results. Management Science, 31, 1499–1508. CrossRefGoogle Scholar
  4. Azen, R., & Budescu, D.V. (2003). The dominance analysis approach for comparing predictors in multiple regression. Psychological Methods, 8, 129–148. PubMedCrossRefGoogle Scholar
  5. Budescu, D.V. (2006). Confidence in aggregation of opinions from multiple sources. In K. Fiedler & P. Juslin (Eds.), Information sampling and adaptive cognition (pp. 327–354). Cambridge: Cambridge University Press. Google Scholar
  6. Budescu, D.V., & Yu, H.T. (2007). Aggregation of opinions based on correlated cues and advisors. Journal of Behavioral Decision Making, 20, 153–177. CrossRefGoogle Scholar
  7. Clemen, R.T., & Winkler, R.L. (1985). Limits for precision and value of information from dependent sources. Operations Research, 33, 427–442. CrossRefGoogle Scholar
  8. Clemen, R.T., & Winkler, R.L. (1986). Combining economic forecasts. Journal of Business and Economic Statistics, 4, 39–46. CrossRefGoogle Scholar
  9. Dawes, R.M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571–582. CrossRefGoogle Scholar
  10. Einhorn, H.J. (1974). Expert Judgment: Some necessary conditions and an example. Journal of Applied Psychology, 59, 562–571. CrossRefGoogle Scholar
  11. Hammond, K.R., & Stewart, T.R. (2001). The essential Brunswik: beginnings, explications, application. London: Oxford University Press. Google Scholar
  12. Hammond, K.R., Wilkins, M.M., & Todd, F.J. (1966). A research paradigm for the study of interpersonal learning. Psychological Bulletin, 65, 221–232. PubMedCrossRefGoogle Scholar
  13. Hogarth, R.M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21, 40–46. CrossRefGoogle Scholar
  14. Hursch, C.J., Hammond, K.R., & Hursch, J.L. (1964). Some methodological considerations in multiple-cue probability studies. Psychological Review, 71, 42–60. PubMedCrossRefGoogle Scholar
  15. Johnson, T.R., Budescu, D.V., & Wallsten, T.S. (2001). Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value. Journal of Behavioral Decision Making, 14, 123–140. CrossRefGoogle Scholar
  16. Miller, S. (2008). Supporting joint human-computer judgment under uncertainty. Unpublished Dissertation at the University of Illinois at Urbana-Champaign. Google Scholar
  17. Morris, P.A. (1986). Comment on Genest and Zideck’s “Combining probability distributions: A critique and annotated bibliography”. Statistical Science, 1, 141–144. CrossRefGoogle Scholar
  18. Shanteau, J. (2001). What does it mean when experts disagree? In E. Salas & G. Klein (Eds.), Linking expertise and naturalistic decision making. Earlbaum: Mahwa. Google Scholar
  19. Schmidt, F.L., Johnson, R.H., & Gugel, J.F. (1978). Utility of policy capturing as an approach to graduate admissions decision making. Applied Psychological Measurement, 2, 345–357. CrossRefGoogle Scholar
  20. Wallsten, T.S., Budescu, D.V., Erev, I., & Diederich, A. (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making, 10, 243–268. CrossRefGoogle Scholar
  21. Wallsten, T.S., & Diederich, A. (2001). Understanding pooled subjective probability estimates. Mathematical Social Sciences, 18, 1–18. CrossRefGoogle Scholar
  22. Weiss, D.J., & Shanteau, J. (2003a). The vice of consensus and the virtue of consistency. In J. Shanteau, P. Johnson, & C. Smith (Eds.), Psychological explorations of competent decision making. Cambridge: Cambridge University Press. Google Scholar
  23. Weiss, D.J., & Shanteau, J. (2003b). Empirical assessment of expertise. Human Factors, 45, 104–116. PubMedCrossRefGoogle Scholar
  24. Winkler, R.L. (1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685. CrossRefGoogle Scholar
  25. Winkler, R.L. (1981). Combining probability distributions from dependent information sources. Management Science, 27, 479–488. CrossRefGoogle Scholar
  26. Winkler, R.L., & Poses, R.M. (1993). Evaluating and combining physician’s probabilities of survival in an intensive care unit. Management Science, 39, 1526–1543. CrossRefGoogle Scholar
  27. Yaniv, I., Choshen-Hillel, S., & Milyavsky, M. (2009). Spurious consensus and opinion revision: Why might people be more confident in their less accurate judgments? Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 558–563. PubMedCrossRefGoogle Scholar

Copyright information

© The Psychometric Society 2009

Authors and Affiliations

  1. 1.Department of PsychologyUniversity of IllinoisChampaignUSA
  2. 2.Department of PsychologyFordham UniversityThe BronxUSA

Personalised recommendations