On the importance of considering heterogeneity in witnesses’ competence levels when reconstructing crimes from multiple witness testimonies

Abstract

Aggregating information across multiple testimonies may improve crime reconstructions. However, different aggregation methods are available, and research on which method is best suited for aggregating multiple observations is lacking. Furthermore, little is known about how variance in the accuracy of individual testimonies impacts the performance of competing aggregation procedures. We investigated the superiority of aggregation-based crime reconstructions involving multiple individual testimonies and whether this superiority varied as a function of the number of witnesses and the degree of heterogeneity in witnesses’ ability to accurately report their observations. Moreover, we examined whether heterogeneity in competence levels differentially affected the relative accuracy of two aggregation procedures: a simple majority rule, which ignores individual differences, and the more complex general Condorcet model (Romney et al., Am Anthropol 88(2):313–338, 1986; Batchelder and Romney, Psychometrika 53(1):71–92, 1988), which takes into account differences in competence between individuals. 121 participants viewed a simulated crime and subsequently answered 128 true/false questions about the crime. We experimentally generated groups of witnesses with homogeneous or heterogeneous competences. Both the majority rule and the general Condorcet model provided more accurate reconstructions of the observed crime than individual testimonies. The superiority of aggregated crime reconstructions involving multiple individual testimonies increased with an increasing number of witnesses. Crime reconstructions were most accurate when competences were heterogeneous and aggregation was based on the general Condorcet model. We argue that a formal aggregation should be considered more often when eyewitness testimonies have to be assessed and that the general Condorcet model provides a good framework for such aggregations.

Appendix: Formalization of the general Condorcet model

In a witness recognition experiment, N witnesses first observe a crime and then make recognition judgments about M statements regarding their observations. In the 2-HTM, responses are modelled as a function of a witness’s competence, D _i , i ∈ {1,…,N}, and the witness’s tendency to guess that a statement is “true”, g _i , when the witness does not know the answer. Each witness is assumed to judge a statement as “true” if the witness believes that a detail has occurred, or as “false” if the witness believes that a detail has not occurred. Thus, “true” responses can be classified either as hits if the statement is true or as false alarms if the statement is false. In the 2-HTM, hits occur either because the witness remembers the relevant fact with probability D _i or because the witness does not remember the relevant fact with probability (1 − D _i ) but guesses correctly with probability g _i. In the 2-HTM, competence and guessing bias are assumed to be constant across questions. Thus, the probability of a hit, H _i , is H _i  = D _i  + (1 − D _i )g _i . False alarms are assumed to occur when the witness does not remember the relevant fact with probability (1 − D _i ) and then incorrectly guesses “true” with probability g _i . Thus, the probability of a false alarm, F _i , can be computed as F _i  = (1 − D _i )g _i . Solving these equations for D _i and g _i yields:

$$D_{i} = H_{i} - F_{i},$$

(1)

and

$$g_{i} = \frac{{F_{i}}}{{\left({1 - H_{i} + F_{i}} \right)}}.$$

(2)

Using the observed hit and false alarm rates as estimates of H _i and F _i , respectively, a witness’s competence and guessing bias can be estimated with Eqs. (1) and (2).

Computing competence and guessing bias in the GCM is more complex because the answer key is unknown. The GCM, therefore, extends the 2-HTM by adding a latent variable, the answer key Z = (Z _k )_1×M , which is a vector of correct responses for items k ∈ {1,…,M}:

$$Z_{k} = \left\{ \begin{array}{ll} 1, & {\rm {\rm{if}}\; {\rm{the}}\; {\rm{correct}}\; {\rm{judgment}}\; {\rm{of}}\; {\rm{item}}}\; k\; {\rm is}\; {\text{``}}{\rm true}{\text{''}}\\ 0, & {\rm {\rm{if}}\; {\rm{the}}\; {\rm{correct}}\; {\rm{judgment}}\; {\rm{of}}\; {\rm{item}}\;} k\; {\rm is}\; {\text{``}}{\rm false}{\text{''}} \end{array}\right.$$

(3)

Further, the GCM (Karabatsos, & Batchelder, 2003; Oravecz et al., 2014) includes another latent variable, the difficulty of item k, δ _k , with 0 < δ _k  < 1. Taking item difficulty into account, Karabatsos and Batchelder (2003) define the probability of witness i knowing the correct response to item k as

$$D_{ik} = \frac{{\theta_{i} \left({1 - \delta_{k}} \right)}}{{\theta_{i} \left({1 - \delta_{k}} \right) + \left({1 - \theta_{i}} \right)\delta_{k}}},$$

(4)

where θ _i denotes the competence of witness i, independent of item difficulty, with 0 < θ _i  < 1.⁶

On the basis of these equations, the GCM defines the probability that witness i correctly recognizes statement k as:

$$p_{ik} = D_{ik}^{{Z_{k}}} + g_{i} \left({1 - D_{ik}} \right)\left({2Z_{k} - 1} \right).$$

(5)

The parameter estimates for the latent parameters competence θ _i , guessing bias g _i , item difficulty δ _k , and the answer key Z _k are determined simultaneously from the response matrix X = (X _ik ) _N×M ,

$$X_{ik} = \left\{ \begin{array}{ll} 1, & { {\rm{if}}\; {\rm{witness}}\; {i}\; {\rm{answers}}\; {\text{``}}{\rm true}{\text{''}}\; {\rm{to}}\; {\rm{item}}}\; k\\ 0, & {{\rm{if}}\; {\rm{witness}}\; i\; {\rm{answers}}\; {\text{``}}{\rm false}{\text{''}}\; {\rm{to}}\; {\rm{item}}\; } k \end{array}\right.$$

(6)

We used the Markov-chain-Monte-Carlo procedure described by Karabatsos and Batchelder (2003) to find parameter estimates that maximize the likelihood function

$$\begin{aligned} L ({\bf{X}|\varOmega} ) = \mathop \prod \limits_{i = 1}^{N} \mathop \prod \limits_{k = 1}^{M} p_{ik}^{{Z_{k} X_{ik} + ({1 - Z_{k}} )({1 - X_{ik}} )}} \times ({1 - p_{ik}} )^{{Z_{k} ({1 - X_{ik}} ) + ({1 - Z_{k}} )X_{ik}}}, \end{aligned}$$

(7)

where $\varOmega = \left\{{\theta_{{\left\langle {i = 1, \ldots, N} \right\rangle}}, g_{{\left\langle {i = 1, \ldots, N} \right\rangle, }} \delta_{{\left\langle {k = 1} \right\rangle}}, Z_{{\left\langle {k = 1, \ldots, M} \right\rangle}}} \right\}$ are the parameters of the GCM. More detailed descriptions of the 2-HTM and the GCM can be found elsewhere (cf. Aßfalg, & Erdfelder, 2012; Batchelder, & Romney, 1986, 1988, 1989; Karabatsos, & Batchelder, 2003; Oravecz et al., 2014; Romney, Batchelder, & Weller, 1987; Romney et al., 1986).