Quantifying the Accuracy of Forensic Examiners in the Absence of a “Gold Standard”

Abstract

This study asked whether latent class modeling methods and multiple ratings of the same cases might permit quantification of the accuracy of forensic assessments. Five evaluators examined 156 redacted court reports concerning criminal defendants who had undergone hospitalization for evaluation or restoration of their adjudicative competence. Evaluators rated each defendant’s Dusky-defined competence to stand trial on a five-point scale as well as each defendant’s understanding of, appreciation of, and reasoning about criminal proceedings. Having multiple ratings per defendant made it possible to estimate accuracy parameters using maximum likelihood and Bayesian approaches, despite the absence of any “gold standard” for the defendants’ true competence status. Evaluators appeared to be very accurate, though this finding should be viewed with caution.

Appendix

Adopting the notation used by Albert (2007), suppose I subjects (i = 1, 2, …, I) undergo assessment by J raters (j = 1, 2, …, J), who assign ordinal ratings k = 1, 2, …, K to each subject. Without loss of generality, let rating k = 1 indicate lowest confidence and k = K indicate highest confidence that a subject has the condition or disorder D of interest (here, incompetence to stand trial). Let Y _i  = (Y _i1, Y _i2, …, Y _iJ )′ be a vector representing ratings made by the J raters for the ith subject. Because J raters could each assign one of K ratings to each subject, each Y _i has J × K possible combinations of elements. The joint distribution of Y _i , expressed as P(Y _i ), the probability of Y _i , is

$$ P ({\user2{Y}}_{{\user2{i}}} )= P ({\user2{Y}}_{{\user2{i}}} |d_{i} = 1 )P (d_{i} = 1 )+ P ({\user2{Y}}_{{\user2{i}}} |d_{i} = 0 )P (d_{i} = 0 ) , $$

(1)

where d _i  = 1 means the ith subject has condition D, d _i  = 0 means the ith subject does not have D, P(d _i  = 1) is the probability or prevalence of D, and P(d _i  = 0) = 1 − P(d _i  = 1).

We would like to model P(Y _i |d _i ) so as to include possible conditional dependence (CD) of ratings, i.e., similarity in raters’ responses attributable to specific characteristics of subjects besides their membership in the D or non-D subgroups that affect how easy or hard their particular cases are. Following Albert, we utilize a probit link function for the parameterization,

$$ \Upphi^{ - 1} \left\{ {P\left( {Y_{ij} \le k|d_{i} ,b_{{d_{i} ,i}} } \right)} \right\} = C_{{d_{i} ,k,j}} + b_{{d_{i} ,i}} $$

(2)

where Φ is the cumulative standard normal distribution function and Φ⁻¹ is its inverse, $C_{{d_{i} ,k,j}}$ are monotonically increasing cut-offs for the jth rater, and $b_{{d_{i} ,i}}$ is a random effect attributable to each subject that characterizes conditional dependence in multiple ratings of that subject.

Notice that $b_{{d_{i}, i}}$ depends on the latent class of each subject—i.e., whether the individual does or does not have D. Following Albert (2007) and Qu et al. (1996), we used the random effect model $b_{{d_{i} , i}} = \sigma_{{d_{i} }} b_{{i}}$, where b _i has a standard normal distribution. Equation 2 thus says that cut-off points demarcating each rater’s classification thresholds reflect the presence (d _i  = 1) or absence (d _i  = 0) of D. However, the probability that the jth rater will assign rating k to the ith subject reflects the ith subject’s state (d _i  = 0 or d _i  = 1), locations of the rater’s particular cut-offs, and peculiarities of the ith subject (which act in common across all raters). We characterize the random effects of the D and non-D populations separately because their cut-offs are not linked (as they would be under the “binormal” ROC model; see Somoza & Mossman, 1991).

In our data set, I = 156, J = 5, and K = 5. Thus, in our CD model, I × J ratings (J raters evaluating I subjects) arise from 2(K − 1)J + 3 = 43 parameters: K − 1 cut-offs for the D subgroup and K − 1 cut-offs for the non-D subgroup for each rater, plus a random effect attributable to each subgroup, plus the prevalence P(d _i  = 1) in the rating set. We sought values for the 43 parameters that would, in combination, be most likely to have generated the 5 × 156 rating matrix. We could then construct individual raters’ ROC graphs using (fpr, tpr) coordinates computed as follows:

$$ fpr_{j,k} = 1 - \Upphi \left( {{\frac{{C_{0,k} }}{{\sqrt {1 + \sigma_{0}^{2} } }}}} \right)\,;\quad tpr_{j,k} = 1 - \Upphi \left( {{\frac{{C_{1,k} }}{{\sqrt {1 + \sigma_{1}^{2} } }}}} \right) $$

(3)

Under a conditional independence (CI) assumption (equivalent to setting $b_{{d_{i} , i}} = 0$), ROC graphs for the five raters could be constructed from estimates of 2(K − 1)J + 1 = 41 parameters.

We estimated the model’s accuracy parameters in two ways. The first approach, standard maximum likelihood estimation (MLE), used GAUSS 3.6 code kindly furnished by Albert and modified for our data. (The modified code, which calls the GAUSS 4.0 maxlik library’s quasi-Newton BFGS optimization algorithm, is available from the first author.) The natural logarithm of the likelihood function, $\ln \,L = \sum\nolimits_{i = 1}^{I} {\ln \,L_{i} }$, is (slightly modifying Albert’s notation)

$$ \ln \,L = \sum\limits_{{i_{1} = 1}}^{K} {\sum\limits_{{i_{2} = 1}}^{K} { \ldots \sum\limits_{{i_{J} = 1}}^{K} {I_{{\{ {\user2{Y}}_{{\user2{i}}} = \,(i_{1} , i_{2} , \ldots ,i_{J} )\} }} \times \ln \left\{ {P\left( {{\user2{Y}}_{{\user2{i}}} = \left( {i_{1} ,\,i_{2} , \ldots ,\,i_{J} } \right)} \right)} \right\},} } } $$

(4)

with P(Y _i ) given by Eq. 1. Knowing (fpr, tpr) coordinates for each rater permitted computation of “trapezoidal” AUCs as overall measures of rater accuracy, with standard errors computed using the method of Hanley and McNeil (1982).

In contrast to MLE, which provides point estimates of the parameter values most likely to have generated the observed data, Bayesian estimation summarizes knowledge of unknown parameters using “posterior” distributions representing the probability that a parameter has a particular value, given the observed data. According to Bayes’ Rule, the posterior probability of a parameter’s value is proportional to the likelihood of observing the data given that parameter value, multiplied by a “prior” probability of the parameter’s value. The likelihood function is dictated by statistical model choice, and is the same construct as in MLE. When a prior is “non-informative” (e.g., P(θ) = c for all θ ∈ [a,b], where [a,b] is an arbitrarily large bounded interval, c is a constant, and θ is a parameter), Bayesian and MLE methods yield similar inferences (Carlin & Louis, 2000). However, in Bayesian estimation, inference is conducted directly on the unknown parameters (or functions thereof, such as AUC), while in MLE, inference is conducted on the data. Hence, only Bayesian estimation allows direct probability statements such as “the probability that the AUC for rater j is between .955 and .973 is 95%.”

Markov chain Monte Carlo (MCMC) methods (Gelfand & Smith, 1990; Geman & Geman, 1984; Metropolis, Rosenbluth, Rosenbluth, Teller, & Teller, 1953) are used to make inferences on posterior distributions for which lack of analytic methods would make Bayes’ Rule intractable. Under mild regularity conditions, a Markov chain converges to a unique invariant or “target” distribution. To use MCMC methods for Bayesian analysis, one constructs the transition kernel so that the target distribution of the resulting Markov chain will be the joint posterior distribution of interest. After discarding input from initial “burn-in” iterations, one can use the remaining draws to make inferences about model parameters. WinBUGS is a free software package that allows specification of a Bayesian model, determines the transition kernel for the Markov chain, and produces draws from the joint posterior distribution of unknown parameters (Lunn et al., 2000).

For our Bayesian analyses, we used minimally informative priors and the same statistical models as in our MLE approach. WinBUGS 1.4.3 ran five parallel MCMC chains; the Brooks–Gelman–Rubin diagnostic (Brooks & Gelman, 1998) indicated convergence after 2000–5000 iterations. We ran each chain for 15,000 iterations and treated each chain’s first 10,000 iterations as “burn-in” values to be discarded, leaving 5 × 5000 = 25,000 draws for inference. Because our WinBUGS code (available from the first author upon request) calculated (fpr, tpr) coordinates and trapezoidal AUCs directly from the MCMC parameter draws, we obtained samples of and made inferences about our accuracy statistics directly from the posterior distributions.