Uncertainty of Qualitative Diagnostic Tests

Abstract

In clinical research gold standard tests for making a diagnosis are often laborious and sometimes impossible. Instead, simple and non-invasive tests are often used. A problem is that these tests have limited sensitivities and specificities. Levels around 50% means that no more information is given than flipping a coin. Levels substantially higher than 50% are commonly accepted as documented proof, that the diagnostic test is valid. However, sensitivity/specificity are estimates from experimental samples, and scientific rigor recommends that with experimental sampling amounts of uncertainty be included. Although the STARD (Standards for Reporting Diagnostic Accuracy) working party recently advised “to include in the estimates of diagnostic accuracy adequate measures of uncertainty, e.g., 95%-confidence intervals” (Bossuyt et al. 2003), so far uncertainty is virtually never assessed in sensitivity/specificity evaluations of clinical diagnostic tests. This is a pity, because calculated levels of uncertainty can be used for statistically testing whether the sensitivity/specificity is significantly larger than 50%. The present chapter uses examples to describe (1) simple methods for calculating standard errors and 95% confidence intervals, and (2) how they can be employed for statistical testing whether the new test is valid. We do hope that this chapter will stimulate clinical investigators to more often assess the uncertainty of the diagnostic tests they apply.

Appendices

Appendix 1

For the calculation of the standard errors (SEs) of sensitivity, specificity and overall-validity we make use of the Gaussian curve assumption in the data.

		Definitive diagnosis (n)
		Yes	No
Result diagnostic test	Yes	a	b
	No	c	d

Sensitivity = a/(a + c) = proportion true positives

Specificity = d/(b + d) = proportion true negatives

1-specificity = b/(b + d)

Proportion of patients with a definitive diagnosis = (a + c)/(a + b + c + d)

Overall validity = (a + d)/(a + b + c + d)

In order to make predictions from these estimates of validity their standard deviations/errors are required. The standard deviation/error (SD/SE) of a proportion can be calculated.

$$ \begin{array}{l}\rm{SD}=\surd \rm{p}\left(1-\rm{p}\right)\rm{where p}=\rm{proportion}.\\ \rm{SE}=\surd \left[\rm{p}\left(1-\rm{p}\right)/\rm{n}\right]\rm{where n}=\rm{sample size}\end{array}$$

where p equals a/(a + c) for the sensitivity. Using the above equations the standard errors can be readily obtained.

$$ \begin{array}{l}{\rm{SE}}_{\rm{sensitivity}}=\surd \rm{ac}/{\left(\rm{a}+\rm{c}\right)}^{3}\\ {\rm{SE}}_{\rm{specificity}}=\surd \rm{db}/{\left(\rm{d}+\rm{b}\right)}^{3}\\ {\rm{SE}}_{1-\rm{specificity}}=\surd \rm{db}/{\left(\rm{d}+\rm{b}\right)}^{3}\\ {\rm{SE}}_{\rm{proportion of patients with a definitive diagnosis}}=\surd \left(\rm{a}+\rm{b}\right)\left(\rm{c}+\rm{d}\right)/{\left(\rm{a}+\rm{b}+\rm{c}+\rm{d}\right)}^{3}\end{array}$$

Appendix 2

The equation of the SE of the overall-validity is less straightforward, but can be obtained using the Bayes’ rule (Berger and Bernerdo 1989) and the delta method (Anonymous 2011). The calculations are given for the purpose of completeness (Var = variance = square root of the standard error; prevalence = proportion of patients with a definitive diagnosis).

$$ \rm{Overall - validity}=\rm{sensitivity}\times \rm{prevalence}+\rm{specificity}\times \left(1-\rm{prevalence}\right)$$

In order to calculate the standard error (SE), we make use of the equation (Var = variance, Cov = covariance)

$$ \rm{Var}\left(\rm{X}+\rm{Y}\right)=\rm{Var}\left(\rm{X}\right)+\rm{Var}\left(\rm{Y}\right)+\rm{2 Cov}\left(\rm{X},\rm{Y}\right)$$

If X = sensitivity × prevalence, and Y = specificity × (1 − prevalence), then the equations can be combined to obtain an equation for the variance of the overall-validity (sens = sensitivity, spec = specificity, prev = prevalence)

$$ {\rm{Var}}_{\rm{overall - validity}}={\rm{Var}}_{\rm{sens}\times \rm{ prev}}+{\rm{Var}}_{\rm{spec}\times (1-\rm{prev})}+{\rm{2 Cov}}_{\rm{sens}\times \rm{prev},\rm{ spec}\times (1-\rm{prev})}$$

The variance of X + Y may according to the delta-method (Anonymous 2011) be approached from:

$$ \rm{Var}\left(\rm{X + Y}\right)={\rm{Y}}^{2}\rm{Var}\left(\rm{X}\right)+{\rm{X}}^{2}\rm{Var}\left(\rm{Y}\right)$$

By combining the equations we will end up finding:

$$ {\rm{Var}}_{\rm{overall - validity}}={\rm{prev}}^{2}\times {\rm{Var}}_{\rm{sens}}+{\left(1-\rm{prev}\right)}^{2}\times {\rm{Var}}_{1-\rm{spec}}+{\left(\rm{sens}-\rm{spec}\right)}^{2}\times {\rm{Var}}_{\rm{prev}}$$

The delta-method describes the variance of natural logarithm (ln) (X) as Var (ln(x)) = Var(x)/x². The approach is sufficiently accurate if the standard errors of prevalence, sensitivity and specificity are small, which is true if samples are not too small. We should add that the delta method is very helpful for the statistical assessment of complex functions like those of standard errors. Second derivatives of parabolas with values similar to those of the complex functions are used to find the best fit parabolas (second order polynomes). Parabolas are easy, and produce a good fit of such complex functions. This methodology has developed tremendously, and terms commonly used for it are the quadratic approximation, eigenvectors, and the delta-method.