Combining Judgements from Correlated Experts

Abstract

When combining the judgements of experts, there are potential correlations between the judgements. This could be as a result of individual experts being subject to the same biases consistently, different experts being subject to the same biases or experts sharing backgrounds and experience. In this chapter we consider the implications of these correlations for both mathematical and behavioural approaches to expert judgement aggregation. We introduce the ideas of mathematical and behavioural aggregation and identify the possible dependencies which may exist in expert judgement elicitation. We describe a number of mathematical methods for expert judgement aggregation, which fall into two broad categories; opinion pooling and Bayesian methods. We qualitatively evaluate which of these methods can incorporate correlations between experts. We also consider behavioural approaches to expert judgement aggregation and the potential effects of correlated experts in this context. We discuss the results of an investigation which evaluated the correlation present in 45 expert judgement studies and the effect of correlations on the resulting aggregated judgements from a subset of the mathematical methods. We see that, in general, Bayesian methods which incorporate correlations outperform mathematical methods which do not.

Appendices

Appendix 1

Suppose we have elicited from an expert the quantiles q ₁, …, q _k corresponding to probabilities p ₁, …, p _k for unknown θ. For example, if p ₁ = 0. 5 then q ₁ would be the expert’s median for θ. Now suppose that we will use an exponential distribution to represent the beliefs of the expert as expressed in the quantiles. For the exponential distribution, quantiles are given by

$$\displaystyle{Q(p_{i},\lambda ) = \dfrac{-\log (1 - p_{i})} {\lambda },}$$

for rate parameter λ which is estimated based on the elicited quantiles. One way to achieve this is to choose λ to minimise the sum of squared differences between the expert’s judgements and the quantiles of the exponential distribution, i.e.,

$$\displaystyle{\hat{\lambda }=\min _{\lambda \in [0,\infty )}\left \{\sum _{i=1}^{k}(q_{ i} - Q(p_{i},\lambda ))^{2}\right \}.}$$

We can find this value analytically by differentiating once and setting the differential equal to zero. Doing so gives

$$\displaystyle{\sum _{i=1}^{k}\left (q_{ i} + \dfrac{\log (1 - p_{i})} {\hat{\lambda }} \right )\left (\dfrac{-\log (1 - p_{i})} {\hat{\lambda }^{2}} \right ) = 0,}$$

and so

$$\displaystyle{\hat{\lambda }= \frac{-\sum _{i=1}^{k}[\log (1 - p_{i})]^{2}} {\sum _{i=1}^{k}q_{i}\log (1 - p_{i})}.}$$

For example, suppose that three quantiles are elicited from an expert, the lower and upper quartiles and the median. Then p ₁ = 0. 25, p ₂ = 0. 5, p ₃ = 0. 75. Suppose that the elicited values are q ₁ = 0. 3, q ₂ = 0. 7, q ₃ = 1. 5. In each case, there is an exact value of λ which satisfies this individual quantile. They are λ ₁ = 0. 96, λ ₂ = 0. 99, λ ₃ = 0. 92. Using the method above, we can find our estimate of λ which approximately satisfies all three quantiles. This is $\hat{\lambda }= 0.94$. Thus, we would say that this expert’s distribution for unknown quantity θ is

$$\displaystyle{\theta \sim \text{Exp}(0.94).}$$

Appendix 2

Number	Study	Seed variables
1	Flange leak	8
2	Crane risk	11
3	Propulsion	13
4	Space debris	18
5	Composite materials	12
6	Option trading	38
7	Risk management	11
8	Groundwater transport	10
9	Acrylo-nitrile	10
10	Dispersion panel TUD	36
11	Dispersion panel TNO	36
12	Dry deposition	24
13	Ammonia Panel	10
14	Sulphur trioxide	10
15	Water pollution	11
16	Environm. panel	28
17	Montserrat volcano	8
18	Campylobacter NL	10
19	Campy Greece	10
20	Oper. risk	16
21	Infosec	10
22	PM25	12
23	Falls ladders	10
24	Dams	11
25	MVOseeds Monserrat follup	5
26	Pilots	10
27	Sete cidades	10
28	TeideMay 05	10
29	VesuvioPisa21Mar05	10
30	Volcrisk	10
31	Sars	10
32	A seed	8
33	Atcep	10
34	Bswaal	8
35	Dcpwwlwl	48
36	Guadeloupe	5
37	Greece NL Carma	10
38	Infoseces	10
39	Oninx	47
40	Pbearlyh	15
41	Return1	15
42	ReturnAfter	31
43	S seed	31
44	Dww exp	15
45	Exp dd	14

Appendix 3

Fig. 9.7

The three quantiles for the aggregated distributions using the Multivariate Normal (M) and Classical methods (C) and the realisation of the corresponding seed variable (R) for all of the seed variables in the space debris study