Bias, underestimation of risk, and loss of statistical power in patient-level analyses of lesion detection

Abstract

Purpose

Sensitivity and the false positive rate are usually defined with the patient as the unit of observation, i.e., the diagnostic test detects or does not detect disease in a patient. For tests designed to find and diagnose lesions, e.g., lung nodules, the usual definitions of sensitivity and specificity may be misleading. In this paper we describe and compare five measures of accuracy of lesion detection.

Methods

The five levels of evaluation considered were patient level without localization, patient level with localization, region of interest (ROI) level without localization, ROI level with localization, and lesion level.

Results

We found that estimators of sensitivity that do not require the reader to correctly locate the lesion overstate sensitivity. Patient-level estimators of sensitivity can be misleading when there is more than one lesion per patient and they reduce study power. Patient-level estimators of the false positive rate can conceal important differences between techniques. Referring clinicians rely on a test’s reported accuracy to both choose the appropriate test and plan management for their patients. If reported sensitivity is overstated, the clinician could choose the test for disease screening, and have false confidence that a negative test represents the true absence of lesions. Similarly, the lower false positive rate associated with patient-level estimators can mislead clinicians about the diagnostic value of the test and consequently that a positive finding is real.

Conclusion

We present clear recommendations for studies assessing and comparing the accuracy of tests tasked with the detection and interpretation of lesions...

Appendix

The sample size calculations in Table 4 follow the methods described in Zhou et al. [2]. For 80% power and 5% type I error rate, a formula for determining the sample size for testing the null hypothesis that two sensitivities are equal, versus the alternative hypothesis that the sensitivities differ, is:

$${N_{\text{D}} = {{\left[ {{\text{2}} \times {\text{Sen}}_{\text{0}} \times \left( {{\text{1}} - {\text{Sen}}_{\text{0}} } \right) \times {\text{7}}{\text{.84}}} \right]} \mathord{\left/ {\vphantom {{\left[ {{\text{2}} \times {\text{Sen}}_{\text{0}} \times \left( {{\text{1}} - {\text{Sen}}_{\text{0}} } \right) \times {\text{7}}{\text{.84}}} \right]} {\left( {{\text{Sen}}_{\text{1}} - {\text{Sen}}_{\text{2}} } \right)^{\text{2}} }}} \right. \kern-\nulldelimiterspace} {\left( {{\text{Sen}}_{\text{1}} - {\text{Sen}}_{\text{2}} } \right)^{\text{2}} }}}$$

where Sen₀ is the sensitivity under the null hypothesis, Sen₁ is the sensitivity of the first technique under the alternative hypothesis, and Sen₂ is the sensitivity of the second technique under the alternative hypothesis. The sample size needed for studies using the level 2 estimators can be estimated by substituting the conjectured values of patient-level sensitivity into this equation.

For studies using lesion-level estimators (or ROI-level estimators) when there is more than one lesion expected per patient (or more than one ROI with a lesion per patient), several modifications to this formula are required. First, instead of patient-level values of sensitivity, we substitute lesion-level (or ROI-level) values of sensitivity. Second, we divide N _D by the number of lesions (or number of ROIs) per patient, then we multiple by the design effect. The design effect [22] is a factor that accounts for the correlation between lesions within the same patient. The design effect equals: ${\text{1}} + \left( {s - {\text{1}}} \right)r$, where s is the number of lesions per patient (or the number of ROIs with lesions per patient), and r is the correlation between lesions within the same patient. If there is no interlesion correlation, then the design effect is 1.0. If there is perfect positive interlesion correlation, then the design effect is s.