The multiple-comparison trap and the Raven’s paradox—perils of using null hypothesis testing in environmental assessment

Abstract

Detecting and quantifying environmental thresholds is frequently an important step in understanding ecological responses to environmental stressors. We discuss two statistical issues often encountered in threshold detection and quantification when statistical null hypothesis testing is used as a main analytical tool. The hidden multiple-comparison trap (leading to a much higher risk of a false detection) and Raven’s paradox (rendering a “detection” meaningless) are often obscured when statistical hypothesis testing is used as part of a more elaborate model, especially models based on computer-intensive methods. Using two examples, we show that the hidden multiple-comparison trap can be exposed using computer simulation to estimate the probability of making a false detection; Raven’s paradox can be avoided by clearly stating the null and alternative hypotheses using scientific terms to substantiate that the rejection of the null is equivalent to proving that the alternative of interest is true. The hidden multiple-comparison trap implies that a null hypothesis testing based on a computer-intensive method should be used with caution. The implication of Raven’s paradox requires that we focus on providing evidence supporting the proposed hypothesis or model, rather than seeking evidence against the frequently irrelevant null hypothesis. These two problems, and many others related to null hypothesis testing, suggest that statistical hypothesis testing should be used only as a component of the body of evidence, perhaps, as the devil’s advocate.