Assessing environmentally significant effects: a better strength-of-evidence than a single P value?

Abstract

Interpreting a P value from a traditional nil hypothesis test as a strength-of-evidence for the existence of an environmentally important difference between two populations of continuous variables (e.g. a chemical concentration) has become commonplace. Yet, there is substantial literature, in many disciplines, that faults this practice. In particular, the hypothesis tested is virtually guaranteed to be false, with the result that P depends far too heavily on the number of samples collected (the ‘sample size’). The end result is a swinging burden-of-proof (permissive at low sample size but precautionary at large sample size). We propose that these tests be reinterpreted as direction detectors (as has been proposed by others, starting from 1960) and that the test’s procedure be performed simultaneously with two types of equivalence tests (one testing that the difference that does exist is contained within an interval of indifference, the other testing that it is beyond that interval—also known as bioequivalence testing). This gives rise to a strength-of-evidence procedure that lends itself to a simple confidence interval interpretation. It is accompanied by a strength-of-evidence matrix that has many desirable features: not only a strong/moderate/dubious/weak categorisation of the results, but also recommendations about the desirability of collecting further data to strengthen findings.

Appendices

Appendices

Appendix 1. Nil hypothesis test result depends on sample size

Consider a survey designed to examine the effect of a marine protected area on populations of a fish, using normal statistical methods (for simplicity, ignoring any influence of data autocorrelation). After collecting n = 50 samples from each area, the mean fish weight per transect inside the MPA is found to be ${\overline{X}}_1=6.1\kern0.75em \mathrm{kg}$, outside it is ${\overline{X}}_2=5.1\kern0.75em \mathrm{kg}$ and the pooled standard deviation of fish sizes inside and outside the MPA is S _p = 4.0 kg. Then, to test the nil hypothesis that there is in fact no difference whatsoever between the means of these two populations (i.e. within the MPA and beyond it), we calculate the test statistic $T=\left|\frac{6.1-5.1}{4.0}\right|\sqrt{\frac{50}{2}}=0.250\times 5=1.250$. Let us now see what T might be for a larger dataset. Bearing in mind that the sample means and variances are unbiased and consistent estimators of their true population values, on average, their values for larger sample sizes would be a little different. Say we have 500 samples at each site and obtain ${\overline{X}}_1=6.2\kern0.75em \mathrm{kg}$, ${\overline{X}}_2=5.3\kern0.75em \mathrm{kg}$ and S _p = 4.2 kg. Then $T=\left|\frac{6.2-5.3}{4.2}\right|\sqrt{\frac{500}{2}}=0.214\times \sqrt{250}=3.388.$ Finally, consider taking 700 samples, obtaining ${\overline{X}}_1=6.0\kern0.75em \mathrm{kg}$, ${\overline{X}}_2=5.1\kern0.75em \mathrm{kg}$ and S _p = 4.3 kg, in which case $T=\left|\frac{6.0-5.1}{4.3}\right|\sqrt{\frac{700}{2}}=0.2093\times \sqrt{350}=3.916.$ The P values for those situations (for example obtained using Excel’s ‘T.DIST.2T’ function) are 0.214, 0.00073 and 0.000094, respectively. At the 5 % significance level, only the last two results would be statistically significant. Having more samples has led to smaller P values—even though the measured effect size happens to have decreased a little (from 0.250 to 0.214 to 0.209). What is more, this pattern will most usually occur when we test a nil hypothesis: P will decrease as n is increased (there will be rare exceptions when the estimated effect size decreases substantially with a larger number of samples, by a factor greater than the square root of the proportional increase in n).

We think that a more meaningful hypothesis to test would be concerned with whether a difference of say 0.5 kg for fish size is large enough to be of environmental significance, so that the equivalence interval limits are ±0.5 kg.

Appendix 2. Procedures are all performed at level α (not α/2)

Under the three-valued logic procedure error risks concern erroneously inferring the direction-of-change in one direction or the other, were such an inference to be made. (The third possible inference—that there are too few data to detect the direction-of-change—is not an error, Jones and Tukey 2000.) In considering the probability of one of these two errors occurring, a critical departure arises from the two-sided nil hypothesis test’s decision rule, which is derived by minimising the risk of committing the type I error (of falsely rejecting a hypothesis).² That rule states that its hypothesis should be rejected if the test statistic cuts off an area of no more than α/2 in either tail of the t distribution—equivalent to examining whether the measured difference in sample means is contained within a 100(1 − α)% confidence interval. In the three-valued logic of the two one-sided TOST approach, the decision rule for each of the two one-sided tests rests on whether the test statistic cuts off an area of no more than α (not α/2) in the appropriate tail of the t distribution. This is equivalent to couching the rule in terms of a 100(1 − 2α)% confidence interval. At first, this may be surprising. Indeed, in the context of bioequivalence tests, Berger and Hsu (1996) noted: ‘The fact that the TOST seemingly corresponds to a 100(1 − 2α)%, not 100(1 − α)%, confidence interval procedure initially caused some concern (Westlake 1976, 1981) … but many authors (e.g. Chow and Shao 1990, and Schuirmann 1989) have defined bioequivalence tests in terms of 100(1–α)% confidence sets’. Indeed, Berger and Hsu (1996), based on earlier material by Berger (1982), present a theorem showing that if each of the two individual TOST tests is performed at level α, the overall test has the same level. Its proof rests on ‘intersection–union’ theory (IUT) (TOST is a simple example of IUT). Importantly, Schuirmann (1996) noted that this finding rests on the requirement that the 100(1 − 2α)% confidence interval is equi-tailed, i.e. is symmetrical about the estimated difference in means, which is the approach adopted herein.