Skip to main content

Why Are Experts Correlated? Decomposing Correlations Between Judges

Abstract

We derive an analytic model of the inter-judge correlation as a function of five underlying parameters. Inter-cue correlation and the number of cues capture our assumptions about the environment, while differentiations between cues, the weights attached to the cues, and (un)reliability describe assumptions about the judges. We study the relative importance of, and interrelations between these five factors with respect to inter-judge correlation. Results highlight the centrality of the inter-cue correlation. We test the model’s predictions with empirical data and illustrate its relevance. For example, we show that, typically, additional judges increase efficacy at a greater rate than additional cues.

This is a preview of subscription content, access via your institution.

References

  • Ariely, D., Au, W.T., Bender, R.H., Budescu, D.V., Dietz, C.B., Gu, H., Wallsten, T.S., & Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6, 130–147.

    PubMed  Article  Google Scholar 

  • Ashton, R.H. (1986). Combining the judgments of experts: How many and which ones? Organizational Behavior and Human Decision Processes, 38, 405–414.

    Article  Google Scholar 

  • Ashton, A.H., & Ashton, R.H. (1985). Aggregating subjective forecasts: Some empirical results. Management Science, 31, 1499–1508.

    Article  Google Scholar 

  • Azen, R., & Budescu, D.V. (2003). The dominance analysis approach for comparing predictors in multiple regression. Psychological Methods, 8, 129–148.

    PubMed  Article  Google Scholar 

  • Budescu, D.V. (2006). Confidence in aggregation of opinions from multiple sources. In K. Fiedler & P. Juslin (Eds.), Information sampling and adaptive cognition (pp. 327–354). Cambridge: Cambridge University Press.

    Google Scholar 

  • Budescu, D.V., & Yu, H.T. (2007). Aggregation of opinions based on correlated cues and advisors. Journal of Behavioral Decision Making, 20, 153–177.

    Article  Google Scholar 

  • Clemen, R.T., & Winkler, R.L. (1985). Limits for precision and value of information from dependent sources. Operations Research, 33, 427–442.

    Article  Google Scholar 

  • Clemen, R.T., & Winkler, R.L. (1986). Combining economic forecasts. Journal of Business and Economic Statistics, 4, 39–46.

    Article  Google Scholar 

  • Dawes, R.M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571–582.

    Article  Google Scholar 

  • Einhorn, H.J. (1974). Expert Judgment: Some necessary conditions and an example. Journal of Applied Psychology, 59, 562–571.

    Article  Google Scholar 

  • Hammond, K.R., & Stewart, T.R. (2001). The essential Brunswik: beginnings, explications, application. London: Oxford University Press.

    Google Scholar 

  • Hammond, K.R., Wilkins, M.M., & Todd, F.J. (1966). A research paradigm for the study of interpersonal learning. Psychological Bulletin, 65, 221–232.

    PubMed  Article  Google Scholar 

  • Hogarth, R.M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21, 40–46.

    Article  Google Scholar 

  • Hursch, C.J., Hammond, K.R., & Hursch, J.L. (1964). Some methodological considerations in multiple-cue probability studies. Psychological Review, 71, 42–60.

    PubMed  Article  Google Scholar 

  • Johnson, T.R., Budescu, D.V., & Wallsten, T.S. (2001). Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value. Journal of Behavioral Decision Making, 14, 123–140.

    Article  Google Scholar 

  • Miller, S. (2008). Supporting joint human-computer judgment under uncertainty. Unpublished Dissertation at the University of Illinois at Urbana-Champaign.

  • Morris, P.A. (1986). Comment on Genest and Zideck’s “Combining probability distributions: A critique and annotated bibliography”. Statistical Science, 1, 141–144.

    Article  Google Scholar 

  • Shanteau, J. (2001). What does it mean when experts disagree? In E. Salas & G. Klein (Eds.), Linking expertise and naturalistic decision making. Earlbaum: Mahwa.

    Google Scholar 

  • Schmidt, F.L., Johnson, R.H., & Gugel, J.F. (1978). Utility of policy capturing as an approach to graduate admissions decision making. Applied Psychological Measurement, 2, 345–357.

    Article  Google Scholar 

  • Wallsten, T.S., Budescu, D.V., Erev, I., & Diederich, A. (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making, 10, 243–268.

    Article  Google Scholar 

  • Wallsten, T.S., & Diederich, A. (2001). Understanding pooled subjective probability estimates. Mathematical Social Sciences, 18, 1–18.

    Article  Google Scholar 

  • Weiss, D.J., & Shanteau, J. (2003a). The vice of consensus and the virtue of consistency. In J. Shanteau, P. Johnson, & C. Smith (Eds.), Psychological explorations of competent decision making. Cambridge: Cambridge University Press.

    Google Scholar 

  • Weiss, D.J., & Shanteau, J. (2003b). Empirical assessment of expertise. Human Factors, 45, 104–116.

    PubMed  Article  Google Scholar 

  • Winkler, R.L. (1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685.

    Article  Google Scholar 

  • Winkler, R.L. (1981). Combining probability distributions from dependent information sources. Management Science, 27, 479–488.

    Article  Google Scholar 

  • Winkler, R.L., & Poses, R.M. (1993). Evaluating and combining physician’s probabilities of survival in an intensive care unit. Management Science, 39, 1526–1543.

    Article  Google Scholar 

  • Yaniv, I., Choshen-Hillel, S., & Milyavsky, M. (2009). Spurious consensus and opinion revision: Why might people be more confident in their less accurate judgments? Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 558–563.

    PubMed  Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Stephen B. Broomell.

Additional information

This work was supported by the National Science Foundation under Awards SES 02-41434 and 03-45925. The first author was supported by a grant from the National Institutes of Health under Ruth L. Kirschstein National Research Service Award PHS 2 T32 MH014257 (“Quantitative Methods for Behavioral Research”) to the University of Illinois at Urbana-Champaign.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Broomell, S.B., Budescu, D.V. Why Are Experts Correlated? Decomposing Correlations Between Judges. Psychometrika 74, 531–553 (2009). https://doi.org/10.1007/s11336-009-9118-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11336-009-9118-z

Keywords

  • information aggregation
  • correlation
  • dependence
  • expert advice