Theoretical Ecology

, Volume 5, Issue 4, pp 611–616 | Cite as

Significance testing in ecological null models

  • Joseph A. VeechEmail author
Brief Communication


In the past decade, the use of null models has become widespread in the testing of ecological theory. Along with increasing usage, null models have also become more complex particularly with regard to tests of significance. Despite the complexity, there are essentially only two distinct ways in which tests of significance are conducted. Direct tests derive a p value directly from the null distribution of a test statistic, as the proportion of the distribution more extreme than the observed value of the test statistic. Indirect tests compare an observed value of a parameter to a null distribution by conducting an additional analysis such as a chi-square test, Kolmogorov–Smirnov test, or regression, although in many cases, this additional step is not necessary. Many kinds of indirect tests require that the null distribution is normal whereas direct tests carry no assumptions about the form of the null distribution. Therefore, when assumptions are violated, indirect tests may have higher type I and II error rates than their counterpart direct tests. A review of 108 null model papers revealed that direct tests were used in 56.5% of studies and indirect tests used in 45.5%. A few studies used both types of test. In general, the randomization algorithms used in most null models should produce normal null distributions, but this could not be confirmed because most studies did not present any description of the null distribution. Researchers should be aware of the differences between direct and indirect tests so as to better use, communicate, and evaluate null models. In many cases, direct tests should be favored for their simplicity and parsimony.


Critical value Randomization Statistical distribution Test statistic Type I error 



I thank Ben Bolker for his very insightful comments on the topic of this manuscript. Sean Connolly, Nick Gotelli, Spyros Sfenthourakis, Werner Ulrich, Diego Vázquez, and two anonymous reviewers also provided helpful comments and suggestions on one or more previous versions of the manuscript.

Supplementary material

12080_2012_159_MOESM1_ESM.doc (110 kb)
ESM 1 (DOC 110 kb)


  1. Ackerly DD, Schwilk DW, Webb CO (2006) Niche evolution and adaptive radiation: testing the order of trait divergence. Ecology 87:S50–S61. doi: 10.1890/0012-9658(2006)87[50:NEAART]2.0.CO;2 PubMedCrossRefGoogle Scholar
  2. Adams DC (2007) Organization of Plethodon salamander communities: guild-based community assembly. Ecology 88:1292–1299. doi: 10.1890/06-0697 PubMedCrossRefGoogle Scholar
  3. Arita HT, Rodriguez P (2004) Local-regional relationships and the geographical distribution of species. Global Ecol Biogeogr 13:15–21. doi: 10.1111/j.1466-882X.2004.00067.x CrossRefGoogle Scholar
  4. Arrington DA, Winemiller KO, Layman CA (2005) Community assembly at the patch scale in a species rich tropical river. Oecologia 144:157–167. doi: 10.1007/s00442-005-0014-7 PubMedCrossRefGoogle Scholar
  5. Azeria ET, Fortin D, Lemaître J, Janssen P, Hébert C, Darveau M, Cumming SG (2009) Fine-scale structure and cross-taxon congruence of bird and beetle assemblages in an old-growth boreal forest mosaic. Global Ecol Biogeogr 18:333–345. doi: 10.1111/j.1466-8238.2009.00454.x CrossRefGoogle Scholar
  6. Bascompte J, Melian CJ (2005) Simple trophic modules for complex food webs. Ecology 86:2868–2873. doi: 10.1890/05-0101 CrossRefGoogle Scholar
  7. Blanchet S, Grenouillet G, Beauchard O, Tedesco PA, Leprieur F, Durr HH, Busson F, Oberdorff T, Brosse S (2010) Non-native species disrupt the worldwide patterns of freshwater fish body size: implications for Bergmann’s rule. Ecol Lett 13:421–431. doi: 10.1111/j.1461-0248.2009.01432.x PubMedCrossRefGoogle Scholar
  8. Bluthgen N, Frund J, Vazquez DP, Menzel F (2008) What do interaction network metrics tell us about specialization and biological traits? Ecology 89:3387–3399. doi: 10.1890/07-2121.1 PubMedCrossRefGoogle Scholar
  9. Burns KC (2006) A simple null model predicts fruit-frugivore interactions in a temperate rainforest. Oikos 115:427–432. doi: 10.1111/j.2006.0030-1299.15068.x CrossRefGoogle Scholar
  10. Burns KC, Zotz G (2010) A hierarchical framework for investigating epiphyte assemblages: networks, meta-communities, and scale. Ecology 91:377–385. doi: 10.1890/08-2004.1 PubMedCrossRefGoogle Scholar
  11. Buston PM, Cant MA (2006) A new perspective on size hierarchies in nature: patterns, causes, and consequences. Oecologia 149:362–372. doi: 10.1007/s00442-006-0442-z PubMedCrossRefGoogle Scholar
  12. Christian KA, Tracy CR, Tracy CR (2006) Evaluating thermoregulation in reptiles: an appropriate null model. Am Nat 168:421–430PubMedCrossRefGoogle Scholar
  13. Connolly SR, Bellwood DR, Hughes TP (2003) Indo-Pacific biodiversity of coral reefs: deviations from a mid-domain model. Ecology 84:2178–2190. doi: 10.1890/02-0254 CrossRefGoogle Scholar
  14. Connor EF, Simberloff D (1979) The assembly of species communities: chance or competition? Ecology 60:1132–1140. doi: 10.2307/1936961 CrossRefGoogle Scholar
  15. Connor EF, Simberloff D (1983) Interspecific competition and species co-occurrence patterns on islands: null models and the evaluation of evidence. Oikos 41:455–465. doi: 10.2307/3544105 CrossRefGoogle Scholar
  16. Cornwell WK, Schwilk DW, Ackerly DD (2006) A trait-based test for habitat filtering: convex hull volume. Ecology 87:1465–1471. doi: 10.1890/0012-9658(2006)87[1465:ATTFHF]2.0.CO;2 PubMedCrossRefGoogle Scholar
  17. Diamond JM (1975) Assembly of species communities. In: Cody ML, Diamond JM (eds) Ecology and evolution of communities. Harvard University Press, CambridgeGoogle Scholar
  18. Diamond JM, Gilpin ME (1982) Examination of the “null” model of Connor and Simberloff for species co-occurrences on islands. Oecologia 52:64–74. doi: 10.1007/BF00349013 CrossRefGoogle Scholar
  19. Efron B, Tibshirani R (1998) An introduction to the bootstrap. CRC Press, New YorkGoogle Scholar
  20. Englund G, Johansson F, Olofsson P, Salonsaari J, Ohman J (2009) Predation leads to assembly rules in fragmented fish communities. Ecol Lett 12:663–671. doi: 10.1111/j.1461-0248.2009.01322.x PubMedCrossRefGoogle Scholar
  21. Farias AA, Jaksic FM (2009) Hierarchical determinants of the functional richness, evenness and divergence of a vertebrate predator assemblage. Oikos 118:591–603. doi: 10.1111/j.1600-0706.2008.16859.x Google Scholar
  22. Fisher RA (1935) The design of experiments. Oliver and Boyd, EdinburghGoogle Scholar
  23. Frank van Veen FJ, Murrell DJ (2005) A simple explanation for universal scaling relations in food webs. Ecology 86:3258–3263. doi: 10.1890/05-0943 CrossRefGoogle Scholar
  24. Fridley JD, Brown RL, Bruno JF (2004) Null models of exotic invasion and scale-dependent patterns of native and exotic species richness. Ecology 85:3215–3222. doi: 10.1890/03-0676 CrossRefGoogle Scholar
  25. Gause GF (1934) The struggle for existence. Williams and Wilkins, BaltimoreCrossRefGoogle Scholar
  26. Gilpin ME, Diamond JM (1982) Factors contributing to non-randomness in species co-occurrences on islands. Oecologia 52:75–84. doi: 10.1007/BF00349014 CrossRefGoogle Scholar
  27. Gotelli NJ, Graves GR (1996) Null models in ecology. Smithsonian Institution Press, Washington, D. CGoogle Scholar
  28. Gotelli NJ, McCabe DJ (2002) Species co-occurrence: a meta-analysis of J. M. Diamond’s assembly rules model. Ecology 83:2091–2096. doi: 10.1890/0012-9658(2002)083[2091:SCOAMA]2.0.CO;2 CrossRefGoogle Scholar
  29. Gotelli NJ, McGill BJ (2006) Null versus neutral models: what’s the difference? Ecography 29:793–800. doi: 10.1111/j.2006.0906-7590.04714.x CrossRefGoogle Scholar
  30. Gotelli NJ, Rohde K (2002) Co-occurrence of ectoparasites of marine fishes: a null model analysis. Ecol Lett 5:86–94. doi: 10.1046/j.1461-0248.2002.00288.x CrossRefGoogle Scholar
  31. Helmus MR, Savage K, Diebel MW, Maxted JT, Ives AR (2007) Separating the determinants of phylogenetic community structure. Ecol Lett 10:917–925. doi: 10.1111/j.1461-0248.2007.01083.x PubMedCrossRefGoogle Scholar
  32. Hoeffding W (1952) The large-sample power of tests based on permutations of observations. Ann Math Stat 23:169–192. doi: 10.1214/aoms/1177729436 CrossRefGoogle Scholar
  33. Horner-Devine MC, Silver JM, Leibold MA, Bohannon BJM, Colwell RK, Fuhrman JA, Green JL, Kuske CR, Martiny JBH, Muyzer G, Ovreas L, Reysenbach A, Smith VH (2007) A comparison of taxon co-occurrence patterns for macro- and microorganisms. Ecology 88:1345–1353. doi: 10.1890/06-0286 PubMedCrossRefGoogle Scholar
  34. Hulme PE (2008) Contrasting alien and native plant species-area relationships: the importance of spatial grain and extent. Global Ecol Biogeogr 17:641–647. doi: 10.1111/j.1466-8238.2008.00404.x CrossRefGoogle Scholar
  35. Hutchinson GE (1959) Homage to Santa Rosalia, or why are there so many kinds of animals? Am Nat 93:145–159CrossRefGoogle Scholar
  36. Ingram T, Shurin JB (2009) Trait-based assembly and phylogenetic structure in northeast Pacific rockfish assemblages. Ecology 90:2444–2453. doi: 10.1890/08-1841.1 PubMedCrossRefGoogle Scholar
  37. Jankowski T, Weyhenmeyer GA (2006) The role of spatial scale and area in determining richness-altitude gradients in Swedish lake phytoplankton communities. Oikos 115:433–442. doi: 10.1111/j.2006.0030-1299.15295.x CrossRefGoogle Scholar
  38. Kembel SW (2009) Disentangling niche and neutral influences on community assembly: assessing the performance of community phylogenetic structure tests. Ecol Lett 12:949–960. doi: 10.1111/j.1461-0248.2009.01354.x PubMedCrossRefGoogle Scholar
  39. Kembel SW, Hubbell SP (2006) The phylogenetic structure of a neotropical forest tree community. Ecology 87:S86–S99. doi: 10.1890/0012-9658(2006)87[86:TPSOAN]2.0.CO;2 PubMedCrossRefGoogle Scholar
  40. Kempthorne O, Doerfler TE (1969) The behavior of some significance tests under experimental randomization. Biometrika 56:231–248. doi: 10.1093/biomet/56.2.231 CrossRefGoogle Scholar
  41. Kohda M, Shibata J, Awata S, Gomagano D, Takeyama T, Hori M, Heg D (2008) Niche differentiation depends on body size in a cichlid fish: a model system of a community structured according to size regularities. J Anim Ecol 77:859–868. doi: 10.1111/j.1365-2656.2008.01414.x PubMedCrossRefGoogle Scholar
  42. Manly BFJ (1998) Randomization, bootstrap and Monte Carlo methods in biology. Chapman & Hall, New YorkGoogle Scholar
  43. Manzaneda AJ, Rey PJ (2009) Assessing ecological specialization of an ant-seed dispersal mutualism through a wide geographic range. Ecology 90:3009–3022. doi: 10.1890/08-2274.1 PubMedCrossRefGoogle Scholar
  44. McCain CM (2004) The mid-domain effect applied to elevational gradients: species richness of small mammals in Costa Rica. J Biogeogr 3:19–31. doi: 10.1046/j.0305-0270.2003.00992.x CrossRefGoogle Scholar
  45. McCain CM (2005) Elevational gradients in diversity of small mammals. Ecology 86:366–372. doi: 10.1890/03-3147 CrossRefGoogle Scholar
  46. Moore JE, Swihart RK (2007) Toward ecologically explicit null models of nestedness. Oecologia 152:763–777. doi: 10.1007/s00442-007-0696-0 PubMedCrossRefGoogle Scholar
  47. Mouillot D, Krasnov BR, Poulin R (2008) High intervality explained by phylogenetic constraints in host-parasite webs. Ecology 89:2043–2051. doi: 10.1890/07-1241.1 PubMedCrossRefGoogle Scholar
  48. Nipperess DA, Beattie AJ (2004) Morphological dispersion of Rhytidoponera assemblages: the importance of spatial scale and null model. Ecology 85:2728–2736. doi: 10.1890/03-0741 CrossRefGoogle Scholar
  49. Okuda T, Noda T, Yamamoto T, Hori M, Nakaoka M (2009) Latitudinal gradients in species richness in assemblages of sessile animals in rocky intertidal zone: mechanisms determining scale-dependent variability. J Anim Ecol 78:328–337. doi: 10.1111/j.1365-2656.2008.01495.x PubMedCrossRefGoogle Scholar
  50. Prado PI, Lewinsohn TM (2004) Compartments in insect-plant associations and their consequences for community structure. J Anim Ecol 73:1168–1178. doi: 10.1111/j.0021-8790.2004.00891.x CrossRefGoogle Scholar
  51. Quinn GP, Keough MJ (2002) Experimental design and data analysis for biologists. Cambridge University Press, NYCrossRefGoogle Scholar
  52. Romano JP (1989) Bootstrap and randomization tests of some nonparametric hypotheses. Ann Stat 17:141–159. doi: 10.1214/aos/1176347007 CrossRefGoogle Scholar
  53. Schoener TW (1984) Size differences among sympatric bird-eating hawks: a worldwide survey. In: Strong DR, Simberloff D, Abele LG, Thistle AB (eds) Ecological communities: conceptual issues and the evidence. Princeton University Press, Princeton, pp 254–281Google Scholar
  54. Simberloff D (1978) Using island biogeographic distributions to determine if colonization is stochastic. Am Nat 112:713–726.CrossRefGoogle Scholar
  55. Simberloff DS (1970) Taxonomic diversity of island biotas. Evolution 24:23–47. doi: 10.2307/2406712 CrossRefGoogle Scholar
  56. Simberloff D, Boecklen W (1981) Santa Rosalia reconsidered: size ratios and competition. Evolution 35:1206–1228. doi: 10.2307/2408133 CrossRefGoogle Scholar
  57. Strong DR, Szyska LA, Simberloff DS (1979) Test of community-wide character displacement against null hypotheses. Evolution 33:897–913. doi: 10.2307/2407653 CrossRefGoogle Scholar
  58. Strong DR, Simberloff D, Abele LG, Thistle AB (1984) Ecological communities: conceptual issues and the evidence. Princeton University Press, PrincetonGoogle Scholar
  59. Swenson NG, Enquist BJ, Pither J, Thompson J, Zimmerman JK (2006) The problem and promise of scale dependency in community phylogenetics. Ecology 87:2418–2424. doi: 10.1890/0012-9658(2006)87[2418:TPAPOS]2.0.CO;2 PubMedCrossRefGoogle Scholar
  60. Thornber CS, Gaines SD (2004) Population demographics in species with biphasic life cycles. Ecology 85:1661–1674. doi: 10.1890/02-4101 CrossRefGoogle Scholar
  61. Tokeshi M (1986) Resource utilization, overlap, and temporal community dynamics: a null model analysis of an epiphytic chironomid community. J Anim Ecol 55:491–506. doi: 10.2307/4733 CrossRefGoogle Scholar
  62. Ulrich W (2004) Species co-occurrences and neutral models: reassessing J. M. Diamond’s assembly rules. Oikos 107:603–609. doi: 10.1111/j.0030-1299.2004.12981.x CrossRefGoogle Scholar
  63. Ulrich W, Gotelli NJ (2007) Null model analysis of species nestedness patterns. Ecology 88:1824–1831. doi: 10.1890/06-1208.1 PubMedCrossRefGoogle Scholar
  64. Veech JA (2000) A null model for detecting nonrandom patterns of species richness along spatial gradients. Ecology 81:1143–1149. doi: 10.1890/0012-9658(2000)081[1143:ANMFDN]2.0.CO;2 CrossRefGoogle Scholar
  65. Veech JA (2006) A probability-based analysis of temporal and spatial co-occurrence in grassland birds. J Biogeogr 33:2145–2153. doi: 10.1111/j.1365-2699.2006.01571.x CrossRefGoogle Scholar
  66. Wilson JB (1995) Null models for assembly rules: the Jack Horner effect is more insidious than the Narcissus effect. Oikos 72:139–144. doi: 10.2307/3546047 CrossRefGoogle Scholar
  67. Wonham MJ, Pachepsky E (2006) A null model of temporal trends in biological invasion records. Ecol Lett 9:663–672. doi: 10.1111/j.1461-0248.2006.00913.x PubMedCrossRefGoogle Scholar
  68. Zimmermann Y, Ramirez SR, Eltz T (2009) Chemical niche differentiation among sympatric species of orchid bees. Ecology 90:2994–3008. doi: 10.1890/08-1858.1 PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2012

Authors and Affiliations

  1. 1.Department of BiologyTexas State UniversitySan MarcosUSA

Personalised recommendations