Validation of null model tests using Neyman–Pearson hypothesis testing theory

Abstract

A long-standing question in ecology is whether interspecific competition affects co-occurrence patterns of species. Null model tests of presence–absence data (NMTPAs) constitute an important approach to address the question, but different tests often give conflicting results when applied to the same data. Neyman–Pearson hypothesis testing theory provides a rigorous and well accepted framework for assessing the validity and optimality of statistical tests. Here, I treat NMTPAs within this framework, and measure the robustness and bias of 72 representative tests. My results indicate that, when restrictive assumptions are met, existing NMTPAs are adequate, but for general testing situations, the use of all existing NMTPAs is unjustified — the tests are nonrobust or biased. For many current applications of NMTPAs, restrictive assumptions appear unmet, so these results illustrate an area in which existing NMTPAs can be improved. In addition to highlighting useful improvements to existing NMTPAs, the results here provide a rigorous framework for developing improved methods.

Appendix

Null hypothesis

Various null hypotheses have been considered for NMTPAs, including null hypotheses that are consistent with species equivalence, site equivalence, and a lack of interspecific interactions. This paper considers the null hypothesis that species are distributed independently of each other (e.g., Connor and Simberloff 1983, p. 463). This null hypothesis is reasonable because interspecific interactions should cause species to occur nonindependently—for instance, with competition, species should be less likely to occur when their competitors are present. If desired, the approach applied here can easily be extended to other null hypotheses.

Let m and n denote the number of species observed and the number of sites sampled, respectively. The sample space, Ω, is the set of binary (0–1) presence–absence matrices with m rows and n columns. Let be the class of all probability measures on , where is the power set of Ω. For b ∈ Ω and i ∈ { 1,...,m} , define the following random variables on : let D _i give the distribution of species i over the n sites; i.e., $\mathbf{D}_{i} :\Omega \to \{ 0,1\} ^{n}$ such that D _i (b) gives row i of b. Let f _i be the distribution of D _i , i.e., f _i is the induced probability measure on { 0,1} ⁿ such that, for any d ∈ { 0,1} ⁿ , $f_{i} (\mathbf{d})=P\{\mathbf{D}_{i}^{-1} (\mathbf{d})\}$. The null hypothesis is:

(1)

The alternative hypothesis is .

Models

The behavior of NMTPAs was investigated under four models (sets of assumptions). For i ∈ { 1,...,m} and j ∈ { 1,...,n} , let X _ij :Ω→{ 0,1} be an indicator variable for the presence of species i at site j; i.e.,

$$ \label{m-1} X_{ij}(\mathbf{b})=\left\{\begin{array}{cc} {0} & {{\rm if\; element\; }i,j{\rm \; of\; }\mathbf{b}{\rm \; is\; 0}} \\ {1} & {{\rm if\; element\; }i,j{\rm \; of\; }\mathbf{b}{\rm \; is\; 1}} \end{array}\right . $$

(2)

Define $p_{ij} \equiv P\{X_{ij}^{-1} (1)\}$, and let P be the m×n matrix whose i,jth entry is p _ij . The first two models are:

(3)

and

(4)

ℳ₂ and ℳ₃ are defined analogously to ℳ₁, except p _ij  = p _kj and p _ij  = p _il , respectively. As noted earlier, ℳ₀ is the only model of these four that is generally realistic.

It is worth noting that models ℳ₁ to ℳ₃ also constrain the possible effects of interspecific interactions. By the definition of conditional probability, for all i,k ∈ { 1,...,m} and j ∈ { 1,...,n},

$$ \begin{array}{lll} \label{m-7} P(X_{ij}=1)= & P(X_{ij}=1|X_{kj}=1)P(X_{kj}=1) \\ & +P(X_{ij}=1|X_{kj}=0)P(X_{kj}=0). \end{array} $$

(5)

Model ℳ₁ asserts that, for all i ∈ { 1,...,m} and j ∈ { 1,...,n}, there exists p ∈ [0,1] such that p _ij  = p. Thus, it follows from (5) that

$$ \begin{array}{lll} \label{m-8} P(X_{ij}=1|&X_{kj}=1) \\ & =\left[p-P(X_{ij}=1|X_{kj}=0)(1-p)\right]/p. \end{array} $$

(6)

An absence of interspecific interactions implies that, for all i,k ∈ { 1,...,m} and j ∈ { 1,...,n}, P(X _ij  = 1|X _kj  = 1) = P(X _ij  = 1|X _kj  = 0). By contrast, competition might cause, for example, P(X ₁₁ = 1|X ₂₁ = 1) = 0.3, P(X ₁₁ = 1| X ₂₁ = 0) = 0.7, P(X ₁₂ = 1|X ₂₂ = 1) = 0.3, and P(X ₁₂ = 1| X ₂₂ = 0) = 0.7. When p = 0.5, this scenario is consis tent with Eq. 6, and hence, it is consistent with ℳ₁. However, if P(X ₁₁ = 1|X ₂₁ = 1) = 0.3, P(X ₁₁ = 1| X ₂₁ = 0) = 0.7, P(X ₁₂ = 1|X ₂₂ = 1) = 0.4, and P(X ₁₂ = 1| X ₂₂ = 0) = 0.8, then there is no p that satisfies (6) for all i,k ∈ { 1,...,m} and j ∈ { 1,...,n}, so even though this scenario is consistent with competition, it is inconsistent with ℳ₁. Similar conditions can be derived for ℳ₂ and ℳ₃. These conditions may be unrealistic in many situations: interspecific interactions may have effects different from those specified by the models.

Hypothesis tests

Let d ₀ and d ₁ be the decisions to accept and reject the null hypothesis (H ₀ ), respectively. The nonrandomized hypothesis test δ:Ω→{ d ₀ ,d ₁ } has critical region $\Omega _{1} \equiv \delta ^{-1} (d_{1} )$ (Bickel et al. 1993, chapter 1; Schervish 1995, chapter 4; Silvey 2003, chapter 6; Lehmann and Romano 2005, chapters 1–3).

Define the following random variables on the probability space :

$$ \label{ht-1} T:\Omega \to {\mathbb N} \mid T(\mathbf{b}){\rm \; gives\; the\; total\; of\; all\; entries\; of\; } \mathbf{b}. $$

(7)

$$ \label{ht-2} \mathbf{R}:\Omega \to {\mathbb N} ^{m} \mid \mathbf{R}(\mathbf{b}) {\rm \; gives\; the\; row\; sums\; of\; } \mathbf{b}. $$

(8)

$$ \label{ht-3} \mathbf{C}:\Omega \to {\mathbb N} ^{n} \mid \mathbf{C}(\mathbf{b}){\rm \; gives\; the\; column\; sums\; of\;} \mathbf{b} . $$

(9)

The tests are combinations of four test statistics, four estimation methods for P, and five conditioning methods. In a few cases, different combinations produce the same test, resulting in 72 rather than 80 distinct tests. The four statistics used (Gotelli 2000) are the checkerboard score (Diamond 1975), number of unique species combinations (“Combo;” Pielou and Pielou 1968), C-score (Stone and Roberts 1990), and V-ratio (Robson 1972), (Schluter 1984), denoted here by W ₁(b) to W ₄(b), respectively. For simplicity, W ₂(b) and W ₄(b) are defined as the negatives of the usual values of Combo and V-ratio, respectively, so all critical regions consist of large values.

The four estimators $\hat{p}_{ij}$ for p _ij (i ∈ { 1,...,m} and j ∈ { 1,...,n} ) follow from assuming that the $\hat{p}_{ij}$ values are (1) all equal, (2) equal within rows and proportional to the observed column totals, (3) equal within columns and proportional to the observed row totals, or (4) proportional to both the observed row and column totals. These are given by: (1) ${\hat{p}_{ij} =T(\mathbf{b})/(mn)}$; (2) ${\hat{p}_{ij} =\sum _{k=1}^{m}X_{kj}(\mathbf{b}) \mathord{\left/ {\vphantom {\hat{p}_{ij} =\sum_{k=1}^{m}X_{kj} (\mathbf{b}) n}} \right. \kern-\nulldelimiterspace}n}$; (3) ${\hat{p}_{ij} =\sum _{l=1}^{n}X_{il}(\mathbf{b}) \mathord{\left/ {\vphantom {\hat{p}_{ij} =\sum_{l=1}^{n}X_{il} (\mathbf{b}) m}} \right. \kern-\nulldelimiterspace}m}$; and (4) $\hat{p}_{ij} =\min \big\{1, \big\{\sum_{k=1}^{m}X_{kj} (\mathbf{b}) \big\} \big\{\sum _{l=1}^{n}X_{il}(\mathbf{b}) \big\} \left/ T(\mathbf{b})\right. \big\}$. For estimators 1 to 4, let $\hat{P}_{1} ,...,\hat{P}_{4}$ be the respective probability measures on , such that $\hat{P}(\mathbf{b})\equiv \prod _{i=1}^{m}\prod _{j=1}^{n}\hat{p}_{ij}^{X_{ij} (\mathbf{b})} (1-\hat{p}_{ij} )^{1-X_{ij} (\mathbf{b})}$.

Five sets of tests Δ₁ ,...,Δ₅ can then be defined by conditioning on different events. Each set contains 16 tests, corresponding to the 16 combinations of W ∈ { W ₁ ,..,W ₄ } and $\hat{P}\in \{ \hat{P}_{1} ,...,\hat{P}_{4} \}$. In some cases, different $\hat{P}\in \{ \hat{P}_{1} ,...,\hat{P}_{4} \}$ give identical tests, resulting in fewer than 16 distinct tests. The five sets are:

$$ \label{ht-4} \Delta _{1} \equiv \big\{ \delta : \Omega _{1} = \big\{ \mathbf{b}:\mathbf{b}\in \Omega ;\; \hat{P}\{W\le W(\mathbf{b})\}\le \alpha \big\} \big\} , $$

(10)

$$ \label{ht-5} \Delta _{2} \equiv \big\{ \delta : \Omega _{1} = \big\{ \mathbf{b}: \mathbf{b}\in \Omega ;\; \,\,\hat{P}\{W\le W(\mathbf{b})|T=T(\mathbf{b})\}\le \alpha \big\} \big\} , $$

(11)

$$ \label{ht-6} \Delta _{3} \equiv \big\{ \delta : \Omega _{1} = \big\{ \mathbf{b}: \mathbf{b}\in \Omega ;\; \,\, \hat{P}\{W\le W(\mathbf{b})|\mathbf{R}=\mathbf{R}(\mathbf{b})\}\le \alpha \big\} \big\} , $$

(12)

$$ \label{ht-7} \Delta _{4} \equiv \big\{ \delta : \Omega _{1} = \big\{ \mathbf{b}: \mathbf{b}\in \Omega ;\; \,\, \hat{P}\{W\le W(\mathbf{b})|\mathbf{C}=\mathbf{C}(\mathbf{b})\}\le \alpha \big\} \big\} , $$

(13)

and

$$ \label{ht-12} \Delta _{5} \equiv \big\{ \delta : \Omega _{1} =\{\mathbf{b}:\mathbf{b}\in \Omega ;\; \,\, \hat{P}\{W\le W(\mathbf{b})|\mathbf{R}=\mathbf{R}(\mathbf{b}),\; \mathbf{C}=\mathbf{C}(\mathbf{b})\}\le \alpha \big\} \big\} . $$

(14)

The motivation for the tests in Δ₁ to Δ₅ is discussed elsewhere (e.g., Connor and Simberloff 1979; Gilpin and Diamond 1982; Gotelli 2000). It is routine to verify that $\delta _{CS} \!=\! (\delta :\delta \in \Delta _{5} ,\; W\!=\!W_{3} ,\; \hat{P}\!=\!\hat{P}_{1} )$; δ _GD  = $(\delta :\delta \in \Delta _{1} ,\; W=W_{1} ,\; \hat{P}=\hat{P}_{4} )$; and δ _G  = (δ:δ ∈ Δ₃, $W=W_{3} ,\; \hat{P}=\hat{P}_{1} )$.

Bias and robustness

Let α be a number between 0 and 1 that gives the maximum tolerable type I error rate. The size of δ under model ℳ is

(15)

If α < α _T , then δ is nonrobust under ℳ. Define

(16)

A test is unbiased under ℳ if β _I  ≥ α _T (Schervish 1995, chapter 4; Casella and Berger 2002, p. 387; Silvey 2003, chapter 6; Lehmann and Romano 2005, chapter 4). The specific aims of this study were threefold: under ℳ₀ to ℳ₃ (1) to find lower bounds for α _T , (2) to find upper bounds for β _I , and (3) to examine whether the bounds depend on m and n.

Numerical procedures

For finding lower bounds for α _T under ℳ _j ∈ { ℳ₀,..., ℳ₃ }, P (Ω₁) was evaluated for several measures P ∈ H ₀ ∩ ℳ _j . The measures and the models to which they belong are given in Fig. 2. The measures correspond to the following ecological scenarios: P ₁ reflects the situation that all species are equally likely to occur at all sites. P ₂ reflects the situation that roughly half of the sites are hospitable while the other half are inhospitable. P ₃ reflects the situation that roughly half the species are unlikely to occur, while the other half are likely to occur. P ₄ to P ₈ reflect other situations of interspecific and spatial heterogeneity. For convenience, the entire set of these measures, which is a subset of H ₀, is denoted $\hat{H}_0$.

Fig. 2

Values of P used in defining $\hat{H}_{0} \equiv {P_{1} ,...,P_{8}}$, and $\hat{H}_{A} \equiv {P_{9} ,...,P_{13}}$. Each square represents the value of P for the given measure. Light and dark shading represents low and high values of p _ij , respectively. Values on the axes indicate row and column numbers. [ ·] denotes the floor function. For i ∈ { 1,...,m} , j ∈ { 1,...,n} , and k ∈ { 9,...,12} , P _k (X _ij  = 1|X _{i − 1,j} = 0) = 1.5p _ij . The measures are elements of the following models: P ₁ ∈ ℳ₀ ∩ ℳ₁ ∩ ℳ₂ ∩ ℳ₃ ; P ₂ ∈ ℳ₀ ∩ ℳ₂ ; P ₃ ∈ ℳ₀ ∩ ℳ₃; P ₄, P ₅,...,P ₈ ∈ ℳ₀; P ₉, P ₁₀,P ₁₁ ∈ ℳ₀ ∩ ℳ₁ ∩ ℳ₂ ∩ ℳ₃ ; P ₁₂ ∈ ℳ₀ ∩ ℳ₂ ; and P ₁₃ ∈ ℳ₀ ∩ ℳ₃

To find upper bounds for β _I under ℳ _j ∈ { ℳ₀,..., ℳ₃ }, P (Ω₁) was evaluated for several choices of P ∈ H _A  ∩ ℳ _j . The measures are defined in Fig. 2, and they correspond to the following ecological scenarios: P ₉ , P ₁₀ , and P ₁₁ reflect the situation that all species are equally likely to occur at all sites. P ₁₂ reflects the situation that roughly half of the sites are less hospitable than the other half. P ₁₃ reflects the situation that roughly half of the species are more likely to occur than the other half. All of these measures reflect a scenario in which each even-numbered species is 1.5 times as likely to occur if the preceding species is absent than if it is present. Similar measures, wherein even-numbered species were twice as likely to occur, were also considered. Such dependence is consistent with widespread competitive interactions. The set of these measures is denoted $\hat{H}_A$.

For the above tests and ℳ _j ∈ { ℳ₀,...,ℳ₃ }, define

(17)

and

(18)

For convenience, when ℳ _j is implied, the subscript j is omitted from $\hat{\alpha }_{T,j}$ and $\hat{\beta }_{I,j}$. Because $\hat{H}_{0} \subseteq H_{0}$, $\hat{\alpha }_{T} \le \alpha _{T}$. Hence, a test is concluded to be nonrobust under ℳ _j if $2\alpha \le \hat{\alpha }_{T}$, a generous criterion because one would not want to make type I errors at a rate of twice the tolerable level. Likewise, because $\hat{H}_A \subseteq H_{A}$, $\beta _{I} \le \hat{\beta }_{I}$. Hence, a test is concluded to be biased under ℳ _j if $2\hat{\beta }_{I} \le \hat{\alpha }_{T}$, as this implies that β _I  < α _T , again a generous criterion.

To find $\hat{\alpha }_{T}$ and $\hat{\beta }_{I}$, for each ℳ _j ∈ { ℳ₀,...,ℳ₃ }, $\delta \in \bigcup _{i=1}^{5}\Delta _{i}$, and $P \in ( \hat{H}_{0} \cup \hat{H}_{A}) \cap$ ℳ _j , I estimated P (Ω₁ ). To do so, I began by setting α = 0.05. For each combination of δ and P , I generated 1,000 matrices from P . I then checked whether each of these matrices was an element of Ω₁ by generating 1,000 random matrices from the appropriate $\hat{P}$. I estimated P (Ω₁ ) by the proportion of matrices from P that were elements of Ω₁ . For tests that conditioned on different statistics but were otherwise identical, I reused empirical distributions. I generated empirical distributions using a Markov chain Monte Carlo/Hastings-Metropolis algorithm (Besag and Clifford 1989; Robert and Casella 1999; Ross 2006, chapter 10). The algorithms were implemented in Visual Basic 6.0 (code available from the author).

To assess whether bias and robustness depend on matrix dimensions, I examined six values of (m,n), the dimensions of the presence–absence matrices in Ω. The dimensions were from presence–absence matrices that have been analyzed previously: 11×25, 17×19, 20×20 (all analyzed in Gotelli 2000), 5×30, 8×9, and 55×7 (all analyzed in Gotelli and McCabe (2002); source data, respectively: Bolger et al. (1991), Meserve and Glanz (1978), and Koopman (1958)).

Analytic results

It can be shown that (1) T is minimally sufficient for P under ℳ₁, (2) R is minimally sufficient for P under ℳ₂, and (3) C is minimally sufficient for P under ℳ₃. Hence, tests that condition (1) on T under ℳ₁, (2) on R under ℳ₂, and (3) on C under ℳ₃ may have Neyman Structure (Cox and Hinkley 2000, p. 135; Lehmann and Romano 2005, p. 115). This structure is indeed present in the following tests under the following models: $\delta :\delta \in \Delta _{2} ,\hat{P}=\hat{P}_{1}$ under ℳ₁; $\delta :\delta \in \Delta _{3} ,\hat{P}=\hat{P}_{1}$ under ℳ₁ and ℳ₃; $\delta :\delta \in \Delta _{4} ,\hat{P}=\hat{P}_{1}$ under ℳ₁ and ℳ₂; $\delta :\delta \in \Delta _{5} ,\hat{P}=\hat{P}_{1}$ under ℳ₁, ℳ₂, and ℳ₃; $\delta :\delta \in \Delta _{3} ,\hat{P}=\hat{P}_{3}$ under ℳ₃, and $\delta :\delta \in \Delta _{4} ,\hat{P}=\hat{P}_{4}$ under ℳ₂. It follows that, under these models, α _T  = α for these tests (Cox and Hinkley 2000, chapter 5).