Why Are Experts Correlated? Decomposing Correlations Between Judges
Rent the article at a discountRent now
* Final gross prices may vary according to local VAT.Get Access
We derive an analytic model of the inter-judge correlation as a function of five underlying parameters. Inter-cue correlation and the number of cues capture our assumptions about the environment, while differentiations between cues, the weights attached to the cues, and (un)reliability describe assumptions about the judges. We study the relative importance of, and interrelations between these five factors with respect to inter-judge correlation. Results highlight the centrality of the inter-cue correlation. We test the model’s predictions with empirical data and illustrate its relevance. For example, we show that, typically, additional judges increase efficacy at a greater rate than additional cues.
- Ariely, D., Au, W.T., Bender, R.H., Budescu, D.V., Dietz, C.B., Gu, H., Wallsten, T.S., & Zauberman, G. (2000). The effects of averaging subjective probability estimates between and within judges. Journal of Experimental Psychology: Applied, 6, 130–147. CrossRef
- Ashton, R.H. (1986). Combining the judgments of experts: How many and which ones? Organizational Behavior and Human Decision Processes, 38, 405–414. CrossRef
- Ashton, A.H., & Ashton, R.H. (1985). Aggregating subjective forecasts: Some empirical results. Management Science, 31, 1499–1508. CrossRef
- Azen, R., & Budescu, D.V. (2003). The dominance analysis approach for comparing predictors in multiple regression. Psychological Methods, 8, 129–148. CrossRef
- Budescu, D.V. (2006). Confidence in aggregation of opinions from multiple sources. In K. Fiedler & P. Juslin (Eds.), Information sampling and adaptive cognition (pp. 327–354). Cambridge: Cambridge University Press.
- Budescu, D.V., & Yu, H.T. (2007). Aggregation of opinions based on correlated cues and advisors. Journal of Behavioral Decision Making, 20, 153–177. CrossRef
- Clemen, R.T., & Winkler, R.L. (1985). Limits for precision and value of information from dependent sources. Operations Research, 33, 427–442. CrossRef
- Clemen, R.T., & Winkler, R.L. (1986). Combining economic forecasts. Journal of Business and Economic Statistics, 4, 39–46. CrossRef
- Dawes, R.M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34, 571–582. CrossRef
- Einhorn, H.J. (1974). Expert Judgment: Some necessary conditions and an example. Journal of Applied Psychology, 59, 562–571. CrossRef
- Hammond, K.R., & Stewart, T.R. (2001). The essential Brunswik: beginnings, explications, application. London: Oxford University Press.
- Hammond, K.R., Wilkins, M.M., & Todd, F.J. (1966). A research paradigm for the study of interpersonal learning. Psychological Bulletin, 65, 221–232. CrossRef
- Hogarth, R.M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21, 40–46. CrossRef
- Hursch, C.J., Hammond, K.R., & Hursch, J.L. (1964). Some methodological considerations in multiple-cue probability studies. Psychological Review, 71, 42–60. CrossRef
- Johnson, T.R., Budescu, D.V., & Wallsten, T.S. (2001). Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value. Journal of Behavioral Decision Making, 14, 123–140. CrossRef
- Miller, S. (2008). Supporting joint human-computer judgment under uncertainty. Unpublished Dissertation at the University of Illinois at Urbana-Champaign.
- Morris, P.A. (1986). Comment on Genest and Zideck’s “Combining probability distributions: A critique and annotated bibliography”. Statistical Science, 1, 141–144. CrossRef
- Shanteau, J. (2001). What does it mean when experts disagree? In E. Salas & G. Klein (Eds.), Linking expertise and naturalistic decision making. Earlbaum: Mahwa.
- Schmidt, F.L., Johnson, R.H., & Gugel, J.F. (1978). Utility of policy capturing as an approach to graduate admissions decision making. Applied Psychological Measurement, 2, 345–357. CrossRef
- Wallsten, T.S., Budescu, D.V., Erev, I., & Diederich, A. (1997). Evaluating and combining subjective probability estimates. Journal of Behavioral Decision Making, 10, 243–268. CrossRef
- Wallsten, T.S., & Diederich, A. (2001). Understanding pooled subjective probability estimates. Mathematical Social Sciences, 18, 1–18. CrossRef
- Weiss, D.J., & Shanteau, J. (2003a). The vice of consensus and the virtue of consistency. In J. Shanteau, P. Johnson, & C. Smith (Eds.), Psychological explorations of competent decision making. Cambridge: Cambridge University Press.
- Weiss, D.J., & Shanteau, J. (2003b). Empirical assessment of expertise. Human Factors, 45, 104–116. CrossRef
- Winkler, R.L. (1971). Probabilistic prediction: Some experimental results. Journal of the American Statistical Association, 66, 675–685. CrossRef
- Winkler, R.L. (1981). Combining probability distributions from dependent information sources. Management Science, 27, 479–488. CrossRef
- Winkler, R.L., & Poses, R.M. (1993). Evaluating and combining physician’s probabilities of survival in an intensive care unit. Management Science, 39, 1526–1543. CrossRef
- Yaniv, I., Choshen-Hillel, S., & Milyavsky, M. (2009). Spurious consensus and opinion revision: Why might people be more confident in their less accurate judgments? Journal of Experimental Psychology: Learning, Memory, and Cognition, 35, 558–563. CrossRef
- Why Are Experts Correlated? Decomposing Correlations Between Judges
Volume 74, Issue 3 , pp 531-553
- Cover Date
- Print ISSN
- Online ISSN
- Additional Links
- information aggregation
- expert advice
- Industry Sectors