Significance testing in ecological null models
- 473 Downloads
In the past decade, the use of null models has become widespread in the testing of ecological theory. Along with increasing usage, null models have also become more complex particularly with regard to tests of significance. Despite the complexity, there are essentially only two distinct ways in which tests of significance are conducted. Direct tests derive a p value directly from the null distribution of a test statistic, as the proportion of the distribution more extreme than the observed value of the test statistic. Indirect tests compare an observed value of a parameter to a null distribution by conducting an additional analysis such as a chi-square test, Kolmogorov–Smirnov test, or regression, although in many cases, this additional step is not necessary. Many kinds of indirect tests require that the null distribution is normal whereas direct tests carry no assumptions about the form of the null distribution. Therefore, when assumptions are violated, indirect tests may have higher type I and II error rates than their counterpart direct tests. A review of 108 null model papers revealed that direct tests were used in 56.5% of studies and indirect tests used in 45.5%. A few studies used both types of test. In general, the randomization algorithms used in most null models should produce normal null distributions, but this could not be confirmed because most studies did not present any description of the null distribution. Researchers should be aware of the differences between direct and indirect tests so as to better use, communicate, and evaluate null models. In many cases, direct tests should be favored for their simplicity and parsimony.
KeywordsCritical value Randomization Statistical distribution Test statistic Type I error
I thank Ben Bolker for his very insightful comments on the topic of this manuscript. Sean Connolly, Nick Gotelli, Spyros Sfenthourakis, Werner Ulrich, Diego Vázquez, and two anonymous reviewers also provided helpful comments and suggestions on one or more previous versions of the manuscript.
- Blanchet S, Grenouillet G, Beauchard O, Tedesco PA, Leprieur F, Durr HH, Busson F, Oberdorff T, Brosse S (2010) Non-native species disrupt the worldwide patterns of freshwater fish body size: implications for Bergmann’s rule. Ecol Lett 13:421–431. doi: 10.1111/j.1461-0248.2009.01432.x PubMedCrossRefGoogle Scholar
- Diamond JM (1975) Assembly of species communities. In: Cody ML, Diamond JM (eds) Ecology and evolution of communities. Harvard University Press, CambridgeGoogle Scholar
- Efron B, Tibshirani R (1998) An introduction to the bootstrap. CRC Press, New YorkGoogle Scholar
- Fisher RA (1935) The design of experiments. Oliver and Boyd, EdinburghGoogle Scholar
- Gotelli NJ, Graves GR (1996) Null models in ecology. Smithsonian Institution Press, Washington, D. CGoogle Scholar
- Horner-Devine MC, Silver JM, Leibold MA, Bohannon BJM, Colwell RK, Fuhrman JA, Green JL, Kuske CR, Martiny JBH, Muyzer G, Ovreas L, Reysenbach A, Smith VH (2007) A comparison of taxon co-occurrence patterns for macro- and microorganisms. Ecology 88:1345–1353. doi: 10.1890/06-0286 PubMedCrossRefGoogle Scholar
- Manly BFJ (1998) Randomization, bootstrap and Monte Carlo methods in biology. Chapman & Hall, New YorkGoogle Scholar
- Schoener TW (1984) Size differences among sympatric bird-eating hawks: a worldwide survey. In: Strong DR, Simberloff D, Abele LG, Thistle AB (eds) Ecological communities: conceptual issues and the evidence. Princeton University Press, Princeton, pp 254–281Google Scholar
- Strong DR, Simberloff D, Abele LG, Thistle AB (1984) Ecological communities: conceptual issues and the evidence. Princeton University Press, PrincetonGoogle Scholar